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CHAPTER ONE 


Overview and 
Value-for-money Audit 
Summaries 





OVERVIEW 
COMMON THEMES 


Our mission is to assist the Legislative Assembly in holding the government and its 
administrators accountable for the quality of the administration’s stewardship of public 
funds and for the achievement of value for money in government operations. In this year’s 
report, we present a number of areas where value for money is not being achieved and 
make recommendations that, if implemented, will assist the Legislature in ensuring that 
government services are being provided in a more cost-effective manner. 


Two themes emerge most strongly from our report this year. They are: 


¢ Far too many concerns noted in our prior audits are not being satisfactorily addressed. 
We found that many of the significant concerns identified in this year’s audits were 
raised by us the last time we audited the program or activity. In some cases, these same 
concerns were raised almost ten years ago. 


¢ All too often, management information systems are not adequate to provide the 
information needed to make sound and prudent business decisions. Over the years, we 
have continually emphasized the importance of better information for decision-making. 
One of senior management's most important responsibilities is to identify what 
information is needed to make informed decisions and to ensure that systems and 
procedures are put in place to provide decision-makers with accurate, useful, and timely 
information. 


Concerns from Previous Audits 


Most of this year’s value-for-money audits examined programs that we have previously 
audited as part of our cyclical VFM-audit approach. Unfortunately, in many of these audits, 
a number of the concerns identified had been raised by us in prior audits, yet they had still 
not been satisfactorily addressed. Here are four examples from this year’s audits: 
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© Court Services, Ministry of the Attorney General—In 1997 we noted that successful 
implementation of a number of initiatives was needed to address the serious backlog of 
court cases and deficiencies in the management of program resources. In this year’s 
audit of court services, we concluded that little progress had been made. As a result, the 
backlog of cases had increased substantially since 1997, efforts to collect millions of 
dollars in outstanding fines continued to be lacking, and the effective administration of 
the courts was still being hampered by the lack of a clear division of responsibility and 
accountability between the Ministry and the Judiciary. 


© Family Responsibility Office, Ministry of Community, Family and Children s Services—This 
Office administers and enforces all court-ordered child and spousal support payments. 
In this year’s audit we concluded that more aggressive and timely enforcement action 
needed to be taken on non-payment of support orders. For example, we found that 
appropriate enforcement action was often not begun until six months after the payment 
deadline had passed, and as long as two years often elapsed between enforcement steps. 
We had expressed similar concerns as far back as 1994 and again in our 1999 audit of 
this program. 


¢ = Children’s Mental Health Services, Ministry of Community, Family and Childrens 
Services—This program funds community-based agencies that provide services to 
children and families of children who have social, emotional, or psychiatric problems. As 
was the case in our last audit of this program in 1997, we found that the Ministry was 
not adequately monitoring and assessing the quality of services being provided by the 
community agencies. As a result, the Ministry cannot be assured that vulnerable 
children in need are receiving the care and assistance they require. 


° Public Health Activity, Ministry of Health—Public health services are jointly funded by 
the Ministry and municipalities and are provided through 37 community-based local 
health units across the province. The services include vaccinations, the prevention of 
communicable diseases, and dealing with emerging public health issues such as the West 
Nile virus and Severe Acute Respiratory Syndrome (SARS). Many of the issues and 
concerns noted during our current audit were also raised by us in 1997. For instance, 
there is still no analysis of the extent to which individuals may be receiving different 
levels of health services depending where in Ontario they live. As well, the required 
medical surveillance procedures for immigrants with inactive tuberculosis are still not 
being adequately monitored and enforced. 


Management Information Systems 
As we have said before, good administration of public funds depends on good decisions 


based on good information. For far too many years we have heard program managers and 
staff express concerns that the information they need is just not available from the systems 
that are in place. Again this year, we encountered a number of situations where the 
management information systems and procedures in place were inadequate to meet the 
information needs of the decision-makers. 
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We believe that ministries’ inability to address this issue is often one of the main contributing 


factors for the lack of effective implementation of our recommendations. This is evident 


from some of the following examples from five of this year’s audits: 


Court Services, Ministry of the Attorney General—The Ministry of the Attorney General, 
the Ministry of Public Safety and Security, and the Judiciary have long recognized the 
need for effective, up-to-date technology for use in management of the courts. 
However, a project to achieve an integrated information system was cancelled after a 
ministry investment of $21 million and six years of effort. With the cancellation of the 
project, there has been little overall improvement to the court’s outdated computer and 
management information systems. 


Family Responsibility Office, Ministry of Community, Family and Children’s Services—In 
our audit we concluded that this Office was in grave danger of failing to meet its 
mandated responsibilities unless it takes a number of steps, including significantly 
improving its information technology and communication systems. As far back as 1994, 
and again in our 1996 follow-up report, the Ministry of the Attorney General, which 
was then responsible for the Office, stated that “the current computer system must be 
replaced”; however, little has been done since that time. 


Children’s Mental Health Services, Ministry of Community, Family and Children s 
Services—In previous audits we found that this program did not have an adequate 
management information system for monitoring financial and service data. In our 
current audit, although a new system had been implemented five years ago, we once 
again found that the system was of limited usefulness, since it did not provide the 
information needed by program staff, and the information it did contain was often 
inaccurate. 


Environet, Ministry of the Environment—In 2000, the Ministry began developing four 
new “Environet” management information systems to strengthen the delivery of its 
environment programs. To date, $17.1 million has been spent, and, while progress is 
being made, the systems do not yet provide ministry staff with the information they need 
to support their monitoring and inspection activities. 


Public Health Activity, Ministry of Health—Controlling disease outbreaks, making 
informed resource allocation decisions, and making public health policies and 
procedures more effective require the ongoing collection, analysis, and interpretation of 
information from across the province. At present, much of the information required is 
either unavailable or is of questionable quality. We expressed concerns about inadequate 
information systems in our 1997 audit, and the Ministry confirmed the need for 
significant improvements in its 2000 Information Technology Strategic Plan. 
Nevertheless, little progress has been made in this area. 
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ACCESS TO INFORMATION 
The Audit Act—the enabling legislation of the Office of the Provincial Auditor—has 


specific provisions with respect to the Auditor’s right to access the information needed to 
fulfill our audit responsibilities. Specifically, section 10 states that we are entitled to have 
access to all books, accounts, financial records, reports, files, and all other papers, things, or 
property necessary to perform our duties under the Act. Clause 12(2)(a) of the Act also 
requires the Auditor to report in the Annual Report on whether, in carrying on the work of 
the Office, the Auditor received all the information and explanations required. 


Until recently, the Office has not had any significant problems accessing the information we 
needed to conduct our audit work. However, in our 2001 Annual Report, we reported an 
instance where the Office did not receive all the information and explanations it required. 
The requested information was ultimately received, albeit after much time and effort had 
been spent. As a result of this incident, Cabinet Office established a protocol dealing with 
the Provincial Auditor’s access to Cabinet-related information. We had hoped that this 
protocol would ensure that such a situation did not reoccur. 


However, during the 2003 audit year we encountered two instances where we did not 
receive timely access to information. One instance related to our audit of court services at 
the Ministry of the Attorney General, and the other related to our audit of science and 
technology at the Ministry of Enterprise, Opportunity and Innovation. While the 
information from the Ministry of the Attorney General was finally received after several 
months of effort, we never did receive full access to the documentation we required from 
the Ministry of Enterprise, Opportunity and Innovation. These two situations are discussed 
in further detail in Chapter Three, sections 3.01 and 3.07, respectively. 


Unfortunately, we found that the protocol on the Provincial Auditor's access to Cabinet 
documents was part of the problem causing the delays. Specifically, we found that ministries 
seemed reluctant to provide certain information to us—much of which used to be provided 
to us on an unhindered basis—because they were apparently not clear on what information 
fell under the protocol. As soon as these problems began surfacing, we started working with 
Cabinet Office and Management Board Secretariat to help establish revised practices, 
particularly with respect to the timely release to us of documentation relating to ministry 
and agency submissions to Cabinet or to its committees such as Management Board of 


Cabinet or SuperBuild. 


Aside from the two instances discussed above, the other ministries and agencies we dealt 
with this year did provide us with all the information, explanations, and other 
documentation we needed. We express our appreciation to the ministry and agency staff 
that we dealt with this year for their co-operation. 
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THE PROVINCE’S FINANCIAL STATEMENTS 
Auditor’s Report 


I am pleased to report that my Auditor's Report on the Province’s financial statements is 
clear of any qualifications or reservations. While my report covers all the financial statements, 
the main focus of the Legislature and the public has historically been on the reported 
surplus or deficit figure for the year. In my opinion, the reported surplus of $117 million 
for the year ended March 31, 2003 is fairly stated. 


Accounting for Major Capital Assets 


In 1997, the Canadian Institute of Chartered Accountants recommended that 
governments adopt the private-sector accounting practice of recognizing capital assets at cost 
in their financial statements and amortizing them to expense over their estimated useful 
lives, rather than expensing them 100% in the year they were acquired. Since then we have 
been encouraging the Ontario government to adopt this approach. 


The government began phasing in this approach for the 2002/03 fiscal year, concentrating 
its efforts on capitalizing land, buildings, and its transportation infrastructure, which 
together account for most of the government's total capital. Accordingly, as at March 31, 
2003, more than $13.9 billion in tangible capital assets owned by the province were for the 
first time reflected in the government's financial statements. The remaining capital assets, 
such as computer systems, vehicles and other equipment, and other smaller-value items, will 
be similarly recorded in subsequent years. We are in agreement with this phased-in 


approach. 


TOWARDS BETTER ACCOUNTABILITY 


Chapter Two of this report provides an update on our long-standing efforts to have our 
proposed amendments to the Audit Act passed by the Legislative Assembly. We also discuss 
two areas where progress is being made in enhancing the accountability of the government 
and government administrators to the Legislative Assembly and the public. 


VALUE-FOR-MONEY AUDIT 
SUMMARIES 


The following are summaries of the value-for-money audits reported on in Chapter Three 


of this Annual Report. 
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3.01 Ministry of the Attorney General 
Court Services 


The Court Services Division (Division) of the Ministry of the Attorney General supports the 
operations of the court system through a network of approximately 250 courthouses and 
approximately 3,500 court support staff. Its functions include providing administrative and 
support services to the Judiciary, preparing enforcement documentation, maintaining court 
records, and collecting fines. 


The Division’s expenditures for the 2002/03 fiscal year were $302 million: $107 million 
was spent on operating the offices of the Judiciary and on salaries and benefits for 
approximately 650 full- and part-time provincially appointed judges; and $195 million was 
spent on administrative and court staffing costs and other expenses required to support the 
operations of courts. In addition, the Ministry spent $35 million on capital projects to 
modernize and improve court buildings. 


In our 1997 audit of what was then the Courts Administration Program, we noted that the 
successful implementation of a number of ongoing initiatives was needed to address the 
serious backlog of cases and deficiencies in the management of program resources. 
However, based on our current audit we concluded that little progress has been made since 
that time. For example: 


¢ Efforts to reduce backlogs have not been effective. At March 2002, approximately 
99,000 criminal charges had been pending at the Ontario Court of Justice for more 
than eight months. This was 39,000 more than in 1998. 


e The Ministry invested approximately $21 million in the Integrated Justice Project to 
develop new information systems. However, the project was terminated five years after 
its establishment with little improvement to the courts’ antiquated computer and 
information systems. 


¢ The lack of ministry effort to collect millions of dollars in outstanding fines weakens the 
credibility of the justice system. 


Other concerns noted during our current audit included: 


* Controls over the planning, contractor selection, and project management for capital 
projects were inadequate. In one case, for example, a contractor was originally hired for 
$52,000 to remove mould, on an emergency basis, at one large courthouse. However, 
further examination of the courthouse revealed the need to address other significant 
building deficiencies. This primary contractor eventually received payments of almost 
$24 million; but in spite of increases in the scope and extent of the work and significant 
cost escalations, competitive quotes were not obtained from other contractors. 


* Numerous significant deficiencies and inconsistencies in the level of security at 
courthouses across the province were noted both by a ministry consultant and during 
our visits to courthouses. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 
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3.02 Ministry of Community, Family and Children’s Services 
Children’s Mental Health Services 


The Children’s Mental Health Services program funds transfer-payment agencies that 
provide services to children and/or the families of children who have social, emotional, or 
behavioural problems or psychiatric disorders. Under provisions of the Child and Family 
Services Act, approximately 250 community-based agencies are funded. The types of services 
offered include assessment, psychiatric therapy, counselling, crisis intervention, and skills 
training and education, as well as residential-based services (mental health services offered in 
a residential setting) to children who require more intensive assistance. 


Children’s Mental Health Services expenditures have increased substantially since the time 
of our last audit, from $213 million in 1996/97 to $315 million in 2002/03. However, 
most of this increase has been spent on several new initiatives in the last two years that in 
most cases provide intensive services to relatively few individuals with complex special needs. 


We concluded that the Ministry was not adequately monitoring and assessing the quality of 
the services provided by the community-based agencies it was funding. As a result, the 
Ministry could not be assured that vulnerable children in need were receiving the care and 
assistance they required. More specifically, we found that the Ministry: 


¢ had not established service quality standards and service evaluation criteria to help 
ministry staff monitor whether or not services were of an acceptable quality and 
represented value for money spent; 


¢ had not established waiting-time standards for access to service that were reasonable and 
commensurate with individual children’s needs, and was not monitoring the extent and 
impact of lengthy waiting times for service; and 


¢ was not receiving or assessing information from agencies about the outcomes of the 
services they were providing. 


We also found that the Ministry’s funding decisions were not based on sufficiently detailed 
and relevant financial and operational information from agencies to ensure that the 
amounts approved were commensurate with the demand for, and level and quality of, 
services to be provided. 


Our findings were of particular concern because many dealt with issues we had previously 
raised in our 1997 audit of the program. Although the Ministry agreed with the 
recommendations in that audit and agreed to implement the necessary corrective action, 
progress has been less than satisfactory. 


We note that the Ministry again agreed with our recommendations in this audit and 
committed to take the necessary corrective action. 
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3.03 Ministry of Community, Family and Children’s Services 

Family Responsibility Office 
Under the authority of the Family Responsibility and Support Arrears Enforcement Act, 1996, 
the Family Responsibility Office (Office) administers and enforces all court-ordered child 
and spousal support in Ontario, as well as court-ordered support in many other jurisdictions 
where the payers are resident in Ontario. The Office also enforces private separation 
agreements that are voluntarily registered with a court and filed with the Office. At the time 
of our audit, the Office was administering approximately 180,600 family-support cases. 


During the 2002/03 fiscal year, the Office collected approximately $561 million from 
support payers and forwarded a similar amount to support recipients. However, as of the 
end of that same year, payment arrears totalled approximately $1.3 billion, which 
represented an 8% increase since our 1999 audit. We also noted that approximately 23,000 
support recipients, whose cases were in arrears totalling over $200 million, were receiving 
provincial social assistance. 


It is our view that unless the Office takes aggressive enforcement action, supported by 
effective case management and significantly improved information technology and 
communications systems, it is in grave danger of failing to meet its mandated responsibilities. 


Our specific findings included the following: 


¢ Since 1994, the number of caseworkers has declined by 20%, whereas the number of 
cases has increased from 126,000 to 180,000, with the result that the average number 
of cases per caseworker has steadily increased. For example, the average number of cases 
with outstanding work items assigned to senior caseworkers has been ranging from 600 
to more than 1,300, averaging 890 cases per caseworker. 


¢ The Office's practice of commencing enforcement action only after being notified by 
recipients of non-payment resulted in unreasonable delays in enforcement. On average, 
seven months elapsed between the time support fell into arrears and the time the Office 
initiated the first enforcement action. 


* More than half the cases in arrears we reviewed had inordinately long gaps—often as 
long as two years—between enforcement actions. 


Staff efforts to enforce support obligations and to provide responsive client services 
continued to be significantly hampered by the Office’s inability to develop and implement 
the necessary improvements to its computer system. Although the Office indicated as far 
back as 1994 that the current computer system must be replaced, the same computer 
system continued to be used even though it cannot provide timely and appropriate 
information to facilitate client service or management of the program. 


We also found that almost 90% of telephone calls made from outside the Greater Toronto 
Area to the Office’s call centre were blocked and therefore not answered. As a result, clients 
had to call repeatedly in order to get through. 


We made a number of recommendations for improvement and received commitments 


from the Office that it would take action to address our concerns. 
ee eee eee 
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3.04 Ministry of Consumer and Business Services 
Policy and Consumer Protection Services Division 


The mandate of the Policy and Consumer Protection Services Division (Division) of the 
Ministry of Consumer and Business Services is to oversee business and other practices in the 
Ontario marketplace. The Marketplace Standards and Services Branch, which accounts for 
approximately two-thirds of the Division’s expenditures, administers various statutes relating 
to consumer protection and business licensing. Its activities include the registering and 
licensing of a number of industries, processing consumer complaints, inspecting businesses 
for compliance with consumer protection acts, and investigating alleged infractions. 


Since 1997, the Ministry has also delegated the administration of several consumer and 
public safety statutes in such sectors as amusement devices, boilers, elevators, fuels, electricity, 
and new home warranties to eight delegated authorities. The delegated authorities are not- 
for-profit corporations that carry out the day-to-day functions of ensuring public safety and 
consumer protection by regulating and monitoring business practices in their industry. 
Nevertheless, the Ministry retains overall responsibility for the outcomes of the delegated 
authorities’ activities in protecting the consumers and the public. The Division's Sector 
Liaison Branch is responsible for overseeing the eight delegated authorities. 


In the 2002/03 fiscal year, the Division had approximately 100 staff and operating 
expenditures of approximately $9.7 million. 


We found that the Marketplace Standards and Services Branch (Branch) did not deploy its 
inspection resources based either on an assessment of risk or on the number of complaints it 
received. For example, while the practices of debt collectors had been the number one 
source of complaints and inquiries received by the Branch, with a total of approximately 
4,000 received in 2001/02, the Branch conducted fewer than 10 inspections of collection 
agencies. In contrast, although the Branch received only eight complaints about theatres 
and video retailers, the Branch devoted 95% of its inspection resources and conducted 
1,600 inspections to check whether video retail stores were operating with a valid licence 
and were selling adult videos only with proper stickers indicating their ratings. 


With respect to the delegated authorities, we noted that: 


© The Ministry had not ensured that data on the outcome of delegated authorities’ 
activities, as reported by them—such as the number of safety-related incidents and the 
number of serious injuries—were reliable. 


° The Ministry had not assessed the sufficiency and appropriateness of the enforcement 
activities undertaken by the delegated authorities in response to identified violations. 


° The Ministry was unable to obtain adequate information about the outcomes and 
activities of the Ontario New Home Warranty Program to assess whether new 
homeowners were being properly protected. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 


i 
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3.05 Ministry of Education 
Curriculum Development and Implementation 


The Education Act gives the Minister of Education broad authority over the “courses of 
study that shall be taught” to the province's 1.4 million elementary and 700,000 secondary 
students in its 4,000 elementary and 800 secondary schools. 


Prior to 1996, school boards had considerable latitude regarding the curriculum that they 
taught. In 1996, however, the Ministry of Education undertook, for the first time, the 
development of a province-wide curriculum. The Ministry began introducing the new 
curriculum in September 1997 and completed its development work with the introduction 
of the grade 12 curriculum in September 2002. The Ministry estimated that the costs of 
developing and implementing the new curriculum between 1996 and January 31, 2003 
were about $488 million. 


We concluded that the process by which the Ministry developed the new curriculum was 
appropriate, and according to most of the educators we interviewed, it resulted in a good- 
quality product that was an improvement over what they had before. 


However, the educators we interviewed expressed concerns regarding the way the 
curriculum was implemented. Their major concern was that the Ministry rushed the 
implementation, with the result that a new curriculum and changes in student assessment 
practices were introduced before appropriate training, textbooks, and other materials were 
readily available. This made the initial years of implementation extremely difficult for 
students and teachers. 


Educators also expressed concerns about the suitability of the new curriculum for weaker 
students. Recent studies and test results indicated that many students are still not succeeding 
under the new curriculum and that many students are entering secondary school without 
the educational foundation required to graduate. 


We also concluded that the Ministry and the school boards we visited did not have sufficient 


and reliable information to, for example: 


¢ measure and report on the extent to which students have learned the new curriculum in 
grades and subjects other than those that have been tested province-wide; 


* measure the extent to which consistency in student assessment has been achieved among 
the province's schools; and 


* identify and prioritize the problems underlying poor student achievement; develop 
viable improvement plans; and track and report results. 


We made recommendations for improving curriculum implementation processes and the 
Ministry committed to taking corrective action. 


ee eeeeeeeeeSFSFSMMMmmm——shFsFeFeFeFeFesesFFFSSSsSSSsS—SsSs 
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3.06 Ministry of Enterprise, Opportunity and Innovation 
Business and Economic Development Activities 


The Ministry of Enterprise, Opportunity and Innovation was created on April 15, 2002, 
with the amalgamation of the science and technology activities of the former Ministry of 
Energy, Science and Technology with the business and economic development activities of 
the former Ministry of Economic Development and Trade. The new Ministry's mandate is 
to foster an Ontario with competitive businesses and a prosperous economy by promoting 
innovation, economic growth, and job creation. In the 2002/03 fiscal year, the Ministry 
employed 300 staff and spent $78.9 million to carry out its business and economic 
development activities. 


We concluded that the Ministry did not have the necessary strategic planning processes and 

information systems to support training institutions in addressing skills shortages and to assist 

Ontario businesses in expanding their export potential. For example, we noted the 

following: 

¢ The Ministry had not evaluated the Strategic Skills Investment program to determine 
whether the program was successful in addressing the current and anticipated skills 
needed to ensure business competitiveness in Ontario and to ensure that students 
obtained employment in their respective areas of training. 


¢ The Ministry had not developed methods to measure the extent to which it had 
achieved its objective of promoting innovation, economic growth, and job creation. 
Instead, the Ministry tended to assess performance by measuring activities; for example, 
it assessed export trade performance by monitoring the number of clients assisted rather 
than by determining whether any increases in export trade had actually occurred as a 
result of ministry activities. 

We found that the Ministry’s advertising and marketing campaign to encourage investment 

in the province was well planned and that appropriate research was carried out to support 

the development of a focused marketing plan. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 


a nn ne EEE EEE SEES! 
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3.07 Ministry of Enterprise, Opportunity and Innovation 
Science and Technology 


The goals of the Ministry of Enterprise, Opportunity and Innovation are to promote 
innovation, economic growth, and job creation. To help achieve these goals, the Ministry 
funded several major science and technology programs and spent $1.3 billion between 
April 1, 1998 and March 31, 2003. The government announced total program 
commitments of $4.3 billion. . 


During our audit, we did not receive adequate access to all the information we requested 
from the Ministry—information that the Ministry must provide to us under section 10 of 
the Audit Act. Such limitations on our access to information prevented us from being able to 
conclude on our audit objectives. Nevertheless, we noted a number of concerns as follows: 


¢ The Ministry has paid the Ontario Innovation Trust $750 million to support the capital 
cost of research in Ontario, but the Ministry receives virtually no information from the 
‘Trust and does not have the required monitoring processes in place to hold the ‘Trust 
accountable for its expenditure of public funds. 


¢ The Innovation Institute of Ontario (IIO)—a non-profit corporation that administers 
the Ontario Research and Development Challenge Fund—often disposed of research- 


proposal assessments without the required written consent from the Ministry. 


¢ We recalculated a sample of 2001/02 Ontario Research Performance Fund grants and 
found that one institution was underpaid by $277,000 and another was overpaid by 
$147,000. We notified the Ministry of these errors, which were subsequently corrected. 


¢ We found that the marks on the Ministry’s summary of the Premier’s Research 
Excellence Awards competition did not agree with the reviewers’ original score-sheet 
marks. Such findings limit the Ministry’s ability to demonstrate the fairness of the 


selection process. 


¢ A review of the minutes from advisory board meetings where research proposals were 
recommended for funding revealed occasions on which a conflict of interest should 
have been declared, but there was no indication in the minutes that a conflict had been 


declared. 


° In July 2000, the Ministry single-sourced the administration of the Ontario Research 
and Development Challenge Fund to the IIO. According to ministry documents, the 
Ministry did not issue a request for proposals (RFP) because once an RFP is out, anyone 
who is eligible must be treated fairly in the process, and problems could arise if a bidder 
was not dealt with fairly. This contradicts the basic principles of government 
procurement. 


The Ministry measured its performance in terms of the expected growth in the value of 
non-government-sponsored research over the next five years. This measure is future 
oriented and does not reflect the ongoing impact of the Ministry’s expenditures to date. 


We made a number of recommendations for improvement and received commitments 


from the Ministry that it would take action to address our concerns. 
net ee eee ee 
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3.08 Ministry of the Environment 
Environet 


The Ministry of the Environment has a broad mandate to restore, protect, and enhance 
Ontario’s environment. It works to ensure cleaner air, water, and land, and healthier 
provincial ecosystems, through a number of acts and associated regulations, including the 
Ontario Water Resources Act and the Environmental Protection Act. 


In 2000, the Ministry developed a new information technology vision and strategy, called 
Environet, to strengthen the delivery of its environmental programs. At the time of our 
audit, the Ministry had spent approximately $17.1 million developing the four Environet 
management information systems we reviewed. 


We concluded that, at the time of our audit, the Ministry’s Environet systems did not 
provide ministry staff with the information needed to support the Ministry’s responsibilities 
of ensuring that drinking water meets regulatory standards, that hazardous waste 
movements are properly controlled, and that all air emissions are monitored and reported 
where required. Our major findings were: 


¢ Total inspection activity is currently at 73% of 1995/96 levels, and inspectors are 
averaging fewer inspections annually. Given the significant increase in the number of 
facilities now covered by new regulations, the Ministry needs to develop a strategy to 
deal with these new facilities. For example, last year inspectors visited only 54 of the 357 
private, drinking water treatment plants and 44 of the 1,119 smaller plants and 
designated facilities. 


¢ Three hundred of 1,476 registered non-municipal waterworks had never submitted 
any test results to the Ministry, and 612 (27%) of registered waterworks had not 
submitted the minimum number of water samples for two of the highest-risk 
substances, EF. coli bacteria and fecal coliform. 


¢ Although exceedances (water samples with unacceptably high concentration levels of 
regulated substances) were only a small proportion of the total samples submitted to the 
Ministry, we did identify 6,725 exceedances since 2001. Of these, 3,181 were Adverse 
Water Quality Incidents, which are more serious exceedances that can affect human 
health. Ministry systems and procedures did not ensure that all Adverse Water Quality 
Incidents were reported and addressed. For example, the Ministry was not aware of 31 
out of 46 Adverse Water Quality Incidents that had occurred at one waterworks. 


¢ Less than 1% of the province’s hazardous waste movements were monitored by the new 
hazardous waste information system. An older, inefficient paper-manifest system was still 
being used to handle almost all transactions because the new system could not accept 
paper submissions. We also noted no evidence of follow-up action on over 5,000 
unauthorized hazardous waste movements flagged by the system. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 
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3.09 Ministry of Health and Long-Term Care 
Public Health Activity 


Under the direction of the Chief Medical Officer of Health, the Ministry’s Public Health 
Branch is responsible for administering the Public Health Activity. The primary legislative 
authority governing the Activity is the Health Protection and Promotion Act. During the 
2001/02 fiscal year, the Ministry provided approximately $222 million to 37 local health 
units, primarily for the delivery of mandatory health programs and services. 


We concluded that the Ministry did not have adequate procedures to ensure that its 
expectations for public health were being met in a cost-effective manner. In particular, we 
were concerned that the Ministry had not analyzed the extent to which individuals received 
differing levels of service depending on where in Ontario they live. For instance, in 2002, 
per capita funding for mandatory health programs and services ranged from approximately 


$23 to $64 among the 37 local health units. 


The Ministry had conducted virtually no regular assessments in the last five years to 
determine whether the health units were complying with the guidelines for mandatory 
programs and services. Such assessments were recommended in the Report of the Walkerton 


Inquiry. We also noted the following: 


¢ Of the 33 local health units reporting information to the Ministry, 13 had conducted 
the required inspections for less than 50% of the high-risk food premises within their 
jurisdiction. Also, 17 health units reported that less than half of the high-risk food 


premises in their jurisdictions had food handlers who had the required training. 


¢ In 2001, local health units inspected only approximately 60% of Ontario’s tobacco 
vendors to verify compliance with the Mandatory Programs and Services Guidelines 


regarding sales to people under the age of 19. 


¢ In 2001, only 65% of individuals identified as requiring medical surveillance for 
tuberculosis were successfully contacted and managed by local health units. Also, we 
were informed that nine local health units would provide a letter for immigrants with 
inactive tuberculosis to verify that the individuals were complying with federal medical 
surveillance requirements. However, the units would provide this letter even if the 
individuals had not had a physical examination and x-ray, which federal guidelines state 
should be administered to such individuals. This places the community at increased risk. 


* The limited information the Ministry had with respect to immunization indicated that 
at least 14% of children had not had all required vaccinations by age seven. 


¢ The Ministry lacked accurate and timely information on communicable diseases and 
immunization, limiting its ability to identify and take any necessary action. 


¢ The Ministry had not yet developed a process to ensure that local health units were 
conducting risk assessments of and taking appropriate action against West Nile virus. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 
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3.10 Ministry of Training, Colleges and Universities 
Ontario Student Assistance Program 


The Ontario Student Assistance Program (OSAP) is a provincially administered and 
federally and provincially funded program that provides needs-based financial assistance to 
full-time students to enable them to attend an approved postsecondary school. The objective 
of OSAP is to help students from lower-income families meet the costs of postsecondary 
education so that all qualified students can have access to postsecondary education. 


Provincial financial assistance to students is provided primarily by loans under the Ontario 
Student Loans (OSL) program. The Ministry also administers the Canada Student Loan 
(CSL) program and the Canada Millennium Bursary (CMB) on behalf of the federal 


gover nment. 


For the 2002/03 fiscal year, provincial OSAP expenditures totalled $356 million. These 
expenditures include default claims on loan guarantees, loan forgiveness grants, interest 
subsidies while students are attending school, interest relief during the repayment stage, and 
various need- and merit-based bursaries and scholarships. 


We concluded that, since our last audit in 1997, the Ministry has taken action to address a 

number of our recommendations and significantly improve the overall administration of the 

program. Notwithstanding, there are several areas where further action is required. In 

particular: 

¢ The Ministry has paid about $2 million more annually in interest costs and risked at 
least $6 million more in annual default costs than it should have because: some loan 
advances to students are paid earlier than necessary; the Ministry has lenient repayment 
policies for loan overpayments; and there was a lack of effective monitoring of academic 
status changes by postsecondary schools, which also caused the Ministry to pay loan 
forgiveness grants to students who were not eligible to receive them. 


¢ Our own analysis to identify instances and patterns of reporting errors or abuse by 
students revealed unlikely circumstances that the Ministry ought to have questioned, 
such as students or parents reporting in their applications an increase of three or more 
dependent children from the previous year. 


¢ The Ministry could further reduce the cost of defaulted student loans by several million 
dollars by making greater use of default management practices successfully employed in 
other jurisdictions. It had also not referred at least 60,000 additional defaulted loans on 
which collection efforts had been unsuccessful to the Canada Customs and Revenue 
Agency, which would have collected the outstanding amounts from any future income 


tax refunds owing to the individuals with the defaulted loans. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take action to address our concerns. 
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CHAPTER TWO 


Towards Better 
Accountability 





Chapter Two of our report is used to address specific issues of accountability and 
governance in government. This year, the chapter focuses on our long-standing efforts to 
have the Audit Act amended and discusses two areas critical to enhancing accountability of 
the government—public reporting on performance and strengthening the modern 
controllership function. 


PROPOSED AMENDMENTS TO THE 
AUDIT ACT 


The Audit Act authorizes the Provincial Auditor to bring to the attention of the Legislature 
any cases where “money was expended without due regard for economy or efficiency” and 
where procedures “to measure and report on the effectiveness of programs” were not 
established. These provisions encompass our value-for-money mandate. 


Since 1989, our Office has pursued amendments to the Audit Act. The primary reason for 
the proposed amendments is to allow the Provincial Auditor to conduct value-for-money 
audits of organizations that receive grants from the province such as community colleges, 
universities, hospitals, and school boards. Given that grants to organizations represent about 
50 % of total government expenditures, we believe our Office needs to have our value-for- 
money audit mandate extended to these expenditures if we are to assist the Legislature in 
fully ensuring that the government is accountable for its stewardship of public funds. The 
current Audit Act allows the Provincial Auditor to examine only the accounting records of 
such organizations, effectively limiting the scope of our work to financial and compliance 
matters and thereby preventing us from conducting full-scope value-for-money audits. 


On April 11, 2003, the then Provincial Auditor wrote the following letter to the Premier of 
Ontario to pursue the Audit Act amendments we believe are necessary. 


Eee 
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Dear Premier Eves: 


The purpose of this letter is to express my Office’s unwavering interest in the 
government taking action on the amendments to the Audit Act unanimously 
approved by the Standing Committee on Public Accounts in November 2002. These 
amendments incorporate the government's commitment to amend the Audit Act as 
outlined in the government’s 2001 Speech from the Throne. The proposed 
amendments are principally designed to permit my Office to better serve Ontario’s 
Legislative Assembly and enhance public accountability of the broader public 
sector. 


As you know, for many years my Office, with the unanimous support and repeated 
recommendations of the Standing Committee on Public Accounts, has been 
requesting amendments to the Audit Act. The most significant amendment that we 
have proposed would provide the Provincial Auditor with the discretionary authority 
to perform full-scope value-for-money audits of organizations that derive a very 
significant portion of their income from provincial grants - such as school boards, 
hospitals, community colleges, and universities. Incidentally, the magnitude of 
provincial grants to these and other organizations is in excess of $30 billion 
annually. In 1996, when you were Minister of Finance, you indicated in a letter to the 
then Chair of the Standing Committee on Public Accounts that you agreed with the 
principles upon which our proposed amendments are based. 


Over the years, there have also been several private members’ bills introduced 
incorporating the main thrust of our proposed amendments as well as those 
proposed by the government in its 2001 Throne Speech. The bill that progressed the 
furthest was Bill 5, which was entitled: An Act to amend the Audit Act to insure 
greater accountability of hospitals universities and colleges, municipalities and other 
organizations which receive grants or other transfer payments from the government 
or agencies of the Crown. Bill 5 was considered during the last session of the 
Legislature and represented the culmination of considerable work carried out over 
more than a decade by my Office and the Standing Committee on Public Accounts, 
including extensive public hearings. In November 2002, the Public Accounts 
Committee unanimously amended and approved Bill 5 and reported it to the 
Assembly for third reading. On December 6, 2002 | wrote to the three House 
Leaders to urge them to consider swift passage of Bill 5 before the House was set 
to rise on December 12, 2002. However, to my disappointment, Bill 5 was never 
called for third reading. 


| would be pleased to arrange a meeting with you or your designate to discuss the 
benefits of making the proposed amendments to the Audit Act, which have received 
public hearings and repeated unanimous endorsements by the Standing Committee 
on Public Accounts. 


Sincerely, 


Erik Peters, FCA 
Provincial Auditor 


Towards Better Accountability 
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We did not receive a response to our letter of April 11. Nevertheless, and in spite of the 
repeated setbacks we have experienced over the years in our efforts to have the Audit Act 
amended, the Office remains committed to pursuing amendments to the Act so that we 
may better serve the Legislative Assembly. 


PUBLIC REPORTING ON 
PERFORMANCE 


Public reporting by government ministries on their performance is intended to inform 
legislators, public servants, and other decision-makers about the extent to which programs 
and services are providing value to the public. Such reporting thus serves as a mechanism 
for accountability and a vehicle to help drive change and focus attention on results as 
opposed to activities and processes. The usefulness and value of performance measurement 
and reporting is reflected in its growing importance to governments around the world in 
recent years. 


Our interest in performance reporting is driven by subclause 12(2)(f)(v) of the Audit Act, 
which requires that the Provincial Auditor report on instances where procedures to measure 
and report on program effectiveness have not been established or were not satisfactory. In 
other words, the Auditor is to report on whether the government is adequately measuring 
and reporting on its performance. When we report on performance, we also make 
recommendations on how performance measures and reporting can be improved. Our 
recommendations might specify, for example, the kinds of performance information that 
decision-makers and stakeholders ought to have available to support good decision-making 
and accountability. 


Over the years, we have found it necessary to report many instances where the government’s 
measuring of and reporting on the efficiency and effectiveness of its programs was 
insufficient. In fact, our Office has been an advocate for improvements in this area as far 
back as 20 years ago. In our 1983 Annual Report, for example, we made several observations 
regarding the need for improved performance reporting by grant recipients such as 
hospitals, school boards, and sports governing bodies. Again this year, in many of the 
Chapter Three sections in this Report, we make a number of recommendations relating to 
the need to improve program performance reporting, 


Notwithstanding our frequent recommendations over the years, we do acknowledge that 
progress is being made. Particularly in recent years, the government has made strides in 
enhancing both the public reporting of performance measures and the use of performance 
measures by program management to help focus efforts and expenditures on the 
achievement of results. 


A very significant step in performance reporting was taken in May 1996 when, in response 


to recommendations of the Ontario Financial Review Commission and the Provincial 


Sr 
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Auditor, the government published Ontario’s first business plans and committed to 
continue publishing the business plans annually. The business plans would include a 
presentation of the results achieved during the year as well as targets, goals, and objectives 
for the next year. 


The process of business planning and public performance reporting that was begun in 
1996 was further improved in 2000, when Management Board Secretariat issued the 
“Business Planning and Allocations Directive” in April and, in December, a companion 
euideline entitled “Performance Measurement In the Business Planning Process—A 
Reference Guide for Ministries.” These documents have provided valuable guidance to 
ministry management and staff as they have endeavoured to enhance their reporting of 
performance, with a particular focus on the outcomes being achieved by significant 
government programs. 


Because of our mandated interest in the subject of performance measurement and the fact 
that the Ontario government has been publishing business plans for eight years, we felt that 
it would be worthwhile and timely to review the guidance provided by Management Board 
Secretariat. As a benchmark, we used the performance-reporting principles recently 
developed and published by the Canadian Comprehensive Auditing Foundation (CCAF). 
CCAF is a national, non-profit research and educational group that has recently developed 
performance-reporting principles for governments in consultation with legislators, senior 
government officials, and legislative auditors across Canada. The principles are contained in 
a CCAF report entitled Reporting Principles—Taking Public Performance to a New Level, 
released in December 2002. The CCAF report explains the importance of reporting 
principles as follows: 


Agreement of principles is an essential step toward confidence in reporting. Principles 
help reporters [those reporting performance] make good judgments and give them 
grounds for confidence that their judgments will be fairly received. They give users 
grounds for confidence that the judgments exercised in reporting are fair, neither arbitrary 
nor self-serving. 


Principles also shape the evolution of reporting: they point the way to what reporting could 

and should be. They start out as ideals, the ceiling that reporting aspires to reach. Over 

time and with growing acceptance, they become standards, the floor below which 

reporting may not sink. 
To date, the principles developed by CCAF have been endorsed by senior public servants 
and the legislative audit community across Canada. For instance, the provinces of British 


Columbia and Saskatchewan have officially adopted CCAF’s performance-reporting 


principles as the model for how they will improve public performance reporting. 


The following chart lists the nine CCAF performance reporting principles and indicates the 
extent to which Management Board’s current guidelines incorporate these principles: 


00 SS 
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Degree of Incorporation of CCAF Principles in ; 
Management Board Secretariat’s Performance Reporting Guidelines 










CCAF Principle = pearpuratee: 






Focus on the few critical aspects of performance. 
Look forward as well as back. 

Explain key risk considerations. 

Explain key capacity considerations. 
Explain other factors critical to performance. 
Integrate financial and non-financial information. 
Provide comparative information. 

Present credible information, fairly interpreted. 


Disclose the basis for reporting. 
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We acknowledge that the CCAF’s performance reporting principles have only recently 
been published. Accordingly, it is not unexpected that some of the recommended principles 
are not included in Ontario’s current performance-reporting guidelines. However, we do 
recommend that over the next few years, as ongoing refinements to the guidelines are made, 
the four principles not yet included be considered in future revisions of the guidelines and 
that ministries be encouraged to implement them as soon as possible. The four principles 
not yet incorporated are described as follows: 


° Explain key risk considerations—Risk relates to the uncertainty of future events and their 
possible impact on the achievement of goals. By identifying key strategic risks and their 
influence on policy choices and performance expectations, a performance report helps 
the reader to relate results achieved to the level of risks accepted. 


¢ Explain capacity considerations—Capacity refers to the ability of an organization to 
achieve a given level of results. Capacity considerations often significantly influence 
decisions about strategic directions, goals, and resource allocations. By explaining 
capacity considerations, a performance report helps the reader to understand and 
interpret an organization’s achievements and plans. 


¢ Explain other factors critical to performance—Other factors that can affect the 
achievement of targeted results include changes in economic, social, demographic, or 
environmental conditions, as well as the actions of other jurisdictions or organizations 
external to the government. The greater the impact these factors have on performance, 
the more important it is to describe that impact. 


° Disclose the basis for reporting—Since performance reports should focus on a few critical 
aspects of performance rather than present volumes of information, it is especially 
important to tell the reader why the reported measures were selected and how they 
relate to the overall strategic direction of the entity. 


eee eee 
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Fully implementing Management Board’s business-planning and performance-reporting 
euidelines across all ministries and improving on these guidelines is an ongoing process. It is 
encouraging to see the progress that has already been made in the public reporting of the 
government's performance. Nevertheless, in these days of constrained resources, increased 
delegation of responsibilities, complex service-delivery relationships, and rapid and constant 
change, the need for clear, credible, and timely performance reporting has never been 
greater. 


ONGOING CONTRIBUTIONS FROM THE 
LEGISLATIVE AUDIT COMMUNITY 


The Canadian Council of Legislative Auditors (CCOLA) is an organization of legislative 
auditors from the federal government and the provinces that meets periodically to share 
ideas and exchange information. CCOLA has been very active in the development of 
CCAP’s performance-reporting principles. 


In 1997, CCOLA established a Performance Reporting and Auditing Group to study and 
make recommendations on ways to advance the quality and usefulness of government 
performance reports. The Group developed a draft set of reporting principles and provided 
advice to CCOLA and CCAF as the CCAF principles were being developed. Recently, the 
Group has focused on the development of audit criteria and programs for interpreting and 
providing assurance on published performance information. 


STRENGTHENING THE MODERN 
CONTROLLERSHIP FUNCTION 


OVERVIEW 


A controllership function is common in most large organizations that maintain accounting 
records and use those records to produce financial information for decision-makers. 
Webster’s dictionary defines a Controller as the chief accounting officer of a business 
enterprise or an institution. In addition to their accounting role, controllers are also 
responsible for establishing a good system of internal control to ensure that information 
produced by the accounting system is timely, relevant, reliable, comparable, and 
understandable. 


Public-sector controllers have historically focused primarily on establishing basic accounting 
systems and financial controls to, for example, ensure that government spending is within 
the levels approved by the Legislature. However, over the last decade there have been a 
number of initiatives at the federal and provincial government level to expand this 
traditional role of the controller. 
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For instance, in 1996, the federal government acknowledged that the modernization of the 
controllership function was key to improving the performance of government. To help 
achieve this, a federal panel known as the Independent Review Panel on Modernization of 
Comptrollership in the Government of Canada was established. The Panel issued a report 
identifying four key elements of modern controllership: 


¢ integrated performance information; 

¢ appropriate control systems; 

¢ asound approach to risk management; and 

¢ ashared set of ethical practices and organizational values. 


The Panel also stated that the single most important change needed was a move to a new 
guiding philosophy for controllership. Essentially, the culture of controllership had to move 
from a “command/control” orientation to a “loose-tight” orientation that would combine a 
strong commitment to centralized standards and values designed to achieve planned results 
with a flexible approach to processes and operating results. 


DEVELOPMENTS IN ONTARIO 


Our Office has long advocated the need for improved information for decision-makers and 
better measurement and reporting of government program performance. We have also 
advocated the need for a strong controllership function as a key means to achieving these 
objectives. For example, in 1993, we played an instrumental role in assisting the 
government's move from producing narrowly focused financial statements that reported on 
the modified cash basis of accounting to more comprehensive accrual-based financial 
statements that consolidated all of the organizations that were accountable to the Legislature 
and owned or controlled by the government. In 1993 we also stressed the need for a 
governmental annual report that would contain not only the audited financial statements 
but also understandable information on the government's fiscal performance. In addition, 
for many years we have recommended that the government adopt the same basis of 
accounting for the Budget and the Estimates as that used in the financial statements and 
that it enhance the Estimates review process by taking performance information into 
consideration during the review process. 


Over the years, the government, largely through the Ministry of Finance, has responded 
positively to many of our recommendations. Significant progress has been made in the area 
of improved financial reporting, in terms of both financial statements and the related 
Budget and Estimates processes. Progress is also being made in developing and 
implementing the key elements of modern controllership in Ontario. Some of the more 
recent initiatives in this regard include the following: 


* In 1999, in response to the global shift to a more results-driven culture in the public 
service, Ontario established a task force to review the controllership function. Particular 
emphasis was placed on reviewing the role of a ministry chief financial officer. The 
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review recognized that a modernized controllership function could improve decision- 
making by establishing processes for the strategic use of financial and performance 
information and for the management of risks within acceptable levels. 


¢ In 1999, both Management Board Secretariat and the Ministry of Finance initiated a 
risk-management review process that ultimately resulted in the development of a Risk 
Management Framework, a “how-to” guide, and a Risk Management Policy. By 
managing risk systematically, managers can increase the likelihood that programs and 
activities will meet their desired objectives. Since the guide was released, a number of 
courses have been provided to train government managers and staff in improving risk 
management practices, controls, and governance processes. 


¢ In 2000, the government implemented a Controllership Capacity Check self- 
assessment tool (the tool had originally been developed for the federal government). The 
tool enabled senior ministry management to assess their controllership practices in seven 
key areas against the best practices of other leading organizations. In 2000, ministries 
completed the self-assessments and developed action plans to improve the controllership 
function in their organizations. 


* In 2000, a Modern Controllership Training Unit was created in the Fiscal and Financial 
Policy Division of the Ministry of Finance to help public servants upgrade their financial 
skills. Courses offered through the unit cover all aspects of modern controllership, 
including risk management, values and ethics, accounting practices and policies, and 
integrated decision-making. Courses are designed for and suited to staff at every level up 
to and including deputy ministers. Since its creation, the unit has provided 
approximately 7,500 days of staff training. 


¢ In 2001, an Intranet Web site was launched to help educate the entire public service on 
modernized controllership. The site provides users with detailed information on modern 
controllership initiatives and government accounting policies and practices, as well as a 
listing of available courses. 


¢ In 2002, the government began the roll-out of its new Integrated Financial Information 
System (IFIS). This government-wide system will replace numerous separate and 
unrelated financial systems that were in use by different ministries across Ontario. IFIS is 
a key component of the government’s modern controllership initiatives and is intended 
to: 


- allow for better estimation of the full cost of government activities, thus improving 
the planning and resource-allocation process; 


-  putall ministries on the same accrual accounting basis, ending the need for manual 
adjustment or reconciliation of data from incompatible financial systems across the 


gover nment; 
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- support the standards for accrual accounting and for capitalization of long-term 
assets recommended by the Canadian Institute of Chartered Accountants’ Public 


Sector Accounting Board; 
- give decision-makers better and faster access to in-year information; and 
- support more timely preparation of the government’s financial statements. 


¢ In 2003, the Ministry of Finance published A Guide to Financial Policies and Practices 
in Ontario. The guide was written in plain English for non-financial decision-makers to 
help them understand the major changes being undertaken in the government’s 
budgeting, planning, accounting, and reporting practices. 


° By 2004, all government accounting systems will operate on the accrual basis. As a 
result, the Budget, the Estimates, the appropriation control system, and the financial 
statements and the Public Accounts for the 2003/04 fiscal year will, for the first time, all 


have been prepared using the same comparable basis of accounting. 


THE WAY AHEAD 


While we are encouraged by the progress made to date in modernizing the controllership 
function, much remains to be done. To realize a full return on its investment in modern 
controllership, the government must successfully make further advances, including the 
following: 


* The culture of the Ontario Public Service must be changed to one where management 
thinks and behaves in “accrual” terms rather than in “cash” terms. Beyond asking, “How 
much money can I spend this year?”, management should also ask, “What government 
resources will I need to deliver this program this year, and are the benefits to the public 
worth these resources?” 


* The government must ensure that individual ministries fully adopt the modern 
controllership approach, in which resources are managed wisely and with due regard 
for economy, efficiency, and effectiveness. Management at ministries must develop 
effective risk-mitigation strategies and a proper internal-control environment and must 
ensure that staff have the information they need to enable them to make informed 
decisions on where and how government resources should be committed. 


¢ The government must ensure that IFIS is implemented smoothly and is both sufficiently 
controlled and sufficiently flexible to produce relevant and reliable government 
financial and operating information for decision-makers. 


These advances will not be possible if controllership is delegated solely to financial specialists. 
Successful modernization of controllership across the hundreds of programs of the Ontario 
government will depend largely on management adopting a “controliership mindset.” In 
other words, a culture in which government managers and staff regularly act in accordance 
with the principles of modern controllership must be developed, under the leadership of 
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the Ministry of Finance and senior program management throughout the government. 
Such a culture will help ensure that risks are managed, resources are used effectively, and 
the effectiveness of the public service is improved. In such a culture, ministries will be better 
able to account to the Legislature and to taxpayers for what they have accomplished with 
the resources entrusted to them. 


Ultimately, a strengthened modern controllership will lead to better decision-making. As we 
have emphasized over the years, good decisions depend on appropriate, reliable, and timely 
information. Such information enables decision-makers to make good choices regarding 
how best to deliver government programs, how to reap the benefits of opportunities, and 
when to take corrective action when problems arise. Thus, ensuring that decision-makers 
have timely access to all the information they need may well be the most significant 
contribution the controllership function can make in the years ahead. 
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CHAPTER THREE 


Reports on 
Value-for-money (VFM) 
Audits 





Our value-for-money audits are intended to examine how well the government’s programs 
and activities are being managed and whether they comply with relevant legislation and 
authorities and, where appropriate, to identify opportunities for improving the economy, 
efficiency, and effectiveness measures of their operations. These audits are conducted under 
subsection 12(2) of the Audit Act, which requires the Office to report on any cases observed 
where money was spent without due regard for economy and efficiency or where 
appropriate procedures were not in place to measure and report on the effectiveness of 
programs. This chapter contains the conclusions, observations, and recommendations for 
the value-for-money audits conducted in the past audit year. 


Because of the size and complexity of the province’s operations and administration, it is not 
practicable nor necessary to audit each program every year. Instead, the Office selects which 
programs and activities to audit on a cyclical basis, so that all major programs and activities 
are audited about every five years. The programs and activities audited this year were 
selected by the Office’s senior management based on various criteria, such as a program’s 
financial impact, its significance to the Legislative Assembly, related issues of public sensitivity 
and safety, and the results of past audits of the program. 


We plan, perform, and report on our value-for-money work in accordance with the 
professional standards for assurance engagements, encompassing value for money and 
compliance, established by the Canadian Institute of Chartered Accountants. 


Before beginning an audit, our staff meet with auditee representatives to discuss the focus of 
the audit. During the audit, staff maintain an ongoing dialogue with the auditee to review 
the progress of the audit and ensure open lines of communication. At the conclusion of the 
audit field work, a draft report is prepared, reviewed internally, and then discussed with the 
auditee. Senior office staff meet with senior management from the ministry or agency to 
discuss the final draft report and to finalize the management responses to our 
recommendations, which are then incorporated into each of the VEM sections. 


ee ee ee ee 
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MINISTRY OF THE ATTORNEY GENERAL 


3.01—Court Services 





BACKGROUND 


In Ontario the court system comprises three distinct courts: the Ontario Court of Justice, 
the Superior Court of Justice, and the Ontario Court of Appeal. 


Approximately 97% of the 560,000 criminal charges tried annually are heard in the 
Ontario Court of Justice, which also tries certain family law cases. The Superior Court of 
Justice tries more serious criminal cases, family law matters, and civil matters including small 
claims. This court may also hear appeals of cases originating in the Ontario Court of Justice. 
The Ontario Court of Appeal hears both criminal and civil appeals. Judges in the Superior 
Court of Justice and the Ontario Court of Appeal are appointed and remunerated by the 
federal government; whereas judges and justices of the peace in the Ontario Court of Justice 


are appointed and paid by the province. We refer to the judges collectively as the Judiciary. 


The Court Services Division (Division) of the Ministry of the Attorney General supports the 
operations of the court system through a network of approximately 250 courthouses and 
approximately 3,500 court support staff. Its primary functions include: 


* providing courtroom staff—clerks, interpreters, and reporters; 


* preparing enforcement documentation and enforcing orders, maintaining court records 


and files, and serving the public and the Bar; 


* providing administrative and support services to the Judiciary, such as trial co- 
ordination, court statistics, caseflow management, and information technology; and 


¢ collecting fines. 


The Division’s expenditures for the 2002/03 fiscal year were $302 million: $107 million for 
operating the offices of the Judiciary and for salaries and benefits for approximately 650 full- 
and part-time provincially appointed judges; and $195 million for all administrative and 
court staffing costs and for other expenses required to support the operation of courts. In 
addition, the Ministry spent $35 million on capital projects to modernize and improve 
court buildings. Revenues pertaining to court services, primarily from fines and court fees, 
were approximately $100 million. 
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AUDIT OBJECTIVES AND SCOPE 


Our audit objectives were to assess whether the Ministry and, where appropriate, the 
Ministry in conjunction with the Judiciary, had adequate systems and procedures in place 
to: 


* ensure that the Division’s resources and capital projects for courts were acquired and 
managed with due regard for economy and efficiency; and 


* measure and report on the effectiveness of the Division’s contribution to providing a fair 
and accessible justice system. 


We identified audit criteria that would be used to address our audit objectives. These were 
reviewed and accepted by ministry senior management. 


Our audit was conducted in accordance with standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants, and accordingly included such tests and other procedures as we 
considered necessary in the circumstances. 


The scope of our audit included interviews with ministry officials, as well as an examination 
of files and documentation at the Ministry’s head office and visits to a number of regional 
offices and courthouses. We also followed up on the recommendations we made in our 
1997 audit of court services. 


We also contacted the Chief Justice of Ontario, on behalf of the Court of Appeal of 
Ontario; the Chief Justice of the Superior Court of Justice; and the Chief Justice of the 
Ontario Court of Justice (collectively referred to as the Chief Justices). The Chief Justices 
provided us with helpful comments and gave us their perspectives on the court system and 


the judicial support services provided by the Ministry. 


We did not rely on the Ministry’s Internal Audit Services Branch to reduce the extent of our 
work because they had not recently completed work within the scope of our audit. 


Our audit fieldwork was substantially completed in March 2003. However, for reasons 
discussed in the following section, it was not until July 2003 that we were able to gain access 
to all the key documents needed to complete our audit. 


ACCESS TO INFORMATION 


The Audit Act requires the Provincial Auditor, in the annual report for each fiscal year, to 
report on whether the Auditor received all the information and explanations required to 
complete the necessary work. Section 10 of the Audit Act states that every ministry of the 
public service: 

...shall furnish the Auditor with such information regarding its powers, duties, activities, 

organization, financial transactions and methods of business as the Auditor from time to 

time requires, and the Auditor shall be given access to all books, accounts, financial 


records, reports, files and all other papers, things or property belonging to or in use by the 
ministry... and necessary to the performance of the duties of the Auditor under this Act. 
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On this audit, we experienced significant delays in receiving several key documents. The 
Ministry had used these documents to obtain approval from the Management Board of 
Cabinet and from the Cabinet Committee on Privatization and SuperBuild (CCOPS) for 
new capital and program initiatives over the previous five years. It took Management Board 
Secretariat and the Ministry from three to five months after our initial requests were made 
in November and December 2002 to provide us with the documents requested. This 
happened despite many subsequent requests on our part to senior management at the 
Ministry. 

In addition, when we did receive the documents, we noted that a number of deletions had 
been made. Following our inquiries regarding these deletions, on May 1, 2003, the 
Ministry informed us that deletions had been made to eight documents as the Ministry had 
a legal question about our statutory right to access all information and documents required 
for audit purposes. After discussions with the Ministry regarding the deletions, this legal 
matter was resolved, and a protocol was developed for dealing with this legal question in the 
future. In July 2003, all documents were provided to us by the Ministry in their entirety. 


OVERALL AUDIT CONCLUSIONS 


In our 1997 audit of what was then the Courts Administration Program, we noted that the 
successful implementation of a number of ongoing initiatives was needed to address the 
serious backlog of cases and deficiencies in the management of program resources. 
However, based on our current audit we concluded that little progress has been made since 
that time. The most significant concerns from our current audit were: 


¢ At March 2002 the Ontario Court of Justice had the highest backlogs of criminal cases 
in 10 years. 


¢ The Integrated Justice Project created to develop the required new information systems 
was terminated five years after its establishment, with virtually no improvement to the 
courts’ antiquated computer and information systems. 


¢ The lack of ministry efforts to collect millions of dollars of overdue fines continued. 


We determined the current status of our 1997 concerns in this audit, and our conclusions 
are outlined in the following table. 
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Current Status of 1997 Audit Concerns 


1997 Audit Concern 


The effective administration of the 
courts was hampered by the lack of a 
clear division of authority and 
responsibility between the Ministry and 
the Judiciary in the management and 
delivery of court services. In 1997, 
negotiations between the Ministry and 
the Judiciary to develop a plan for 
reform were in progress. 





Serious backlogs of criminal cases 
existed. New information systems and 
approaches were needed to help reduce 
backlogs. 


Current Status 


The Ministry was unable to achieve 
consensus with the Judiciary, and 
negotiations were therefore terminated. 


Efforts to reduce backlogs have not been 
effective. Caseloads and pending charges in 
the Ontario Court of Justice had increased 
significantly. At March 31, 2002, there were 
approximately 99,000 criminal charges that 
had been pending for more than eight 
months. This was 39,000 more than in 1998. 





Courts’ information systems are 
antiquated, largely paper driven, and in 
need of automation. The Integrated 
Justice Project was initiated in 1996 to 
develop technological solutions and 
allow exchange of data among the 
various users in the justice system. 


Adequate financial information systems 
to assess whether results were 
achieved at appropriate costs did not 
exist. 





The Ministry, in conjunction with the 
Judiciary, needed to measure and 
report on its effectiveness in providing 
courts that are fair and accessible. 


The Ministry did not ensure that the 
collection of overdue fines was 
vigorously pursued. 





After the Ministry spent approximately 

$21 million on the Integrated Justice Project, 
the project was terminated. As a result, no 
substantial progress was made in introducing 
new information technology to the courts. 





The Ministry still did not have any financial 
information systems, at either the corporate 
or regional level, that would allow 
management to monitor how cost effectively 
court services were delivered. 


Performance measures were not established 
to measure and report to the public on the 
effectiveness and efficiency of courts, as 
demonstrated by, for example, waiting times 
for trials and court costs. 


The Ministry made little effort to collect 
outstanding fines. At one point in 2002, more 
than two-and-a-half years had elapsed 
before the Ministry transferred outstanding 
fines to Management Board Secretariat’s 
Collection Management Unit. The lack of 
collection efforts weakens the credibility of 
the justice system. 


Our current audit also identified the following additional concerns: 


* Controls over the planning, contractor selection, and project management for capital 
projects were inadequate. For example, a contractor was originally hired for $52,000 to 
remove mould at one large courthouse on an emergency basis. However, further 
examination of the courthouse also revealed the need to address significant building 
deficiencies on an urgent basis. This primary contractor eventually received payments of 
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almost $24 million; but in spite of increases in the scope and extent of the work and 
significant cost escalations, competitive quotes were not obtained from other contractors. 
In another example, the estimated cost of $30 million to construct a new courthouse 
was about $9 million more than forecasted and 40% higher than for other similar 
projects. In yet another example, contrary to the Management Board of Cabinet 
directive requiring the use of the Ontario Realty Corporation and without seeking 
competitive tenders, the Ministry contracted directly with a project management 
company for construction work at a courthouse and paid this company $187,000, even 
though the Ministry was aware the work had not been started. 


¢ Aconsultant’s review of security risks completed in January 2003 identified numerous 
significant deficiencies at courthouses across the province. We also noted significant 
inconsistencies in the level of security during our own visits to courthouses. 


DETAILED AUDIT OBSERVATIONS 
ADMINISTRATIVE STRUCTURE OF THE COURTS 


The Judiciary is independent of the administrative and legislative arms of the government. 
As part of its adjudication function, the Judiciary is responsible for the conduct of 
proceedings within its courtrooms. It directs the operation of courts, such as determining 
the dates of court sittings, the scheduling of cases, and the assigning of judges. While the 
Judiciary controls the use of court resources, the Ministry decides on court budgets, staffing 
decisions, courthouse capital projects, and the number of judges. It has long been 
acknowledged that this division of responsibilities can only be successful if there is a clearly 
defined accountability structure and a clear division of authority and responsibility between 
the Judiciary and the Ministry. However, the need for such a structure and division of 
authority and responsibility has not been successfully dealt with over the years. 


In 1995, a joint Ministry and Judiciary study, entitled Czvil Justice Review—First Report, 
concluded: “The justice system can no longer function effectively in Ontario unless a single 
authority, with clear lines of responsibility and accountability, is established to deal with all 
administrative, financial and budgetary, and operational matters relating to court 
administration in the Province.” 


In our 1997 Annual Report, we recommended that in order for the justice system to 
function more effectively, the Ministry and the Judiciary needed to ensure that there be a 
reform of the management of court services that clearly established accountability and 
responsibility for achieving desired results. At that time, discussions were in progress to 
address this issue and a Framework Agreement was reached between the Ministry and the 
Judiciary to develop a plan for the reform of the management of court services in Ontario. 
Subsequently, however, the Ministry was unable to achieve consensus with the Judiciary, and 
the project was terminated. 
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During our current audit, the Ministry informed us that it was committed to “relationship 
building” with the Judiciary, in particular with respect to the management of operational 
issues. [he aim was to encourage positive joint involvement in decision-making. 


Although the Chief Justices considered the Court Services Division to be responsive to the 
needs of the Judiciary, they also cited areas where the Ministry’s support services were not 
meeting the needs of the courts. In particular, they cited the issues of staffing, training, and 
security. We also noted several long-standing concerns in the judicial system that have not 
been resolved and might be more effectively dealt with by an improved administrative and 
accountability arrangement between the Judiciary and the Ministry. These concerns 
included backlogs, performance standards and measures, information technology, and court 
security, and are discussed in the sections that follow. 


Recommendation | 


To help ensure that the justice system functions effectively and to improve the 
stewardship of funds provided to the courts, the Ministry and Judiciary should 
improve their administrative and management procedures by establishing: 








e aprocess of greater co-operation in decision-making that addresses long- 
standing concerns; 


e abetter structure of courts administration with greater accountability for 
achieving desired results such as reducing case backlogs. 


Ministry Response 


The Ministry has undertaken to work with the Judiciary to build good 
relationships at all levels and to explore joint management of, and decision- 
making about, existing institutional challenges and ongoing operational issues. 
Joint working groups have been established to include representatives of both 
courts, as well as the bar and relevant agencies, to support ongoing 
identification and implementation of measures to aces long-standing 
challenges such as backlogs. 


Efforts to bring about joint management with the Judiciary are occurring ona 
number of fronts. For example: 


¢ Members of the Judiciary have been involved since January 2002 in the 
Court Services Division’s development of service standards and a five-year 
business plan for the Division. Further consultation will be conducted 
annually to review and update the plan. The establishment of service 
standards as part of the planning process will support greater 
accountability for the Division’s delivery of court administration services. 

¢ Written standards for judicial support have now been established for the 


Superior Court, and standards for the Ontario Court will be completed this 
fiscal year. 
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¢ The Chief Justice of the Ontario Court has asked the Division to participate 


in a process to develop a complement model for the Court. 





CASE BACKLOGS 


The success of the judicial system is measured by its ability to fairly resolve disputes in a 
timely manner. In 1992, the Supreme Court of Canada provided a guideline of eight to 
10 months as a reasonable period of time to allow for cases going to trial. The Ministry 
maintains statistics to show how many outstanding charges are older than eight months. 


In 1993 and 1997, we reported that serious backlogs existed for criminal cases. As 
illustrated in the following chart, backlogs of pending charges continued to grow in the 
Ontario Court of Justice, which handled the majority of criminal cases. 


Ontario Court of Justice 
Five-Year Summary of Average Age of Criminal Charges Pending 
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Source of data: Ministry of the Attorney General 


Over the five-year period from 1997/98 to 2001/02, the total number of pending charges 
at the Ontario Court of Justice grew by 37%, and the number of criminal charges in the 
courts with an average age of more than eight months increased by approximately 65% or 
39,000. The problem was more serious at certain courthouses, particularly those in large 
urban centres such as Toronto, Ottawa, and Brampton. At some of these locations, it took 
up to 12 months to schedule cases requiring a full day of court time—these were typically 
the more serious cases. 


The Superior Court of Justice was also experiencing significant backlogs in its criminal and 
family courts. Conversely, the Ontario Court of Appeal indicated it had been successful in 
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eliminating its backlogs through better control over trial scheduling and screening and 
through termination of appeals with no merits. The average delay in hearings for civil 
appeals had been reduced from 30 months to five months; for criminal appeals, it had been 


reduced from six to 12 months to three to four months. 


Many of the factors that contribute to delays in cases being heard are beyond the control of 
the Ministry and the Judiciary. Examples of these factors include the increasing complexity 
and length of criminal cases, the unreadiness of the parties, and the growing number of self- 
represented litigants in the courts, leading to more lengthy proceedings. 


Furthermore, the actions and decisions of each party involved in a trial—including the 
police, Crown and defence attorneys, and the litigants—all have an impact on the number 
of court appearances and the length of a trial. This in turn affects the utilization of 
courtrooms and the ability of the Judiciary to dispose of cases on a timely basis. 


In this regard, we noted that the average number of court appearances for a case to be 
resolved in the Ontario Court of Justice increased more than 20%, from 5.89 appearances 
in 1996/97 to 7.24 appearances in 2001/02. The Ministry could not provide specific 
reasons for this increase, but it recognized that the increase placed additional demands on 
the resources of courts and judges. 


Over the years, the Ministry and Judiciary have introduced a number of measures to 
address the issue of backlogs. For example, since our 1997 Annual Report, the number of 
judges has increased by approximately 5%, case management procedures have been 
established for certain types of cases to monitor unreasonable delays, and mandatory 
mediation procedures have been used to encourage parties in civil cases to resolve disputes 
without costly and lengthy court hearings. For courts with serious backlogs, blitzes— 
through the appointment and reallocation of judges and Crown attorneys—have been used 
to deal with more cases. 


However, the measures introduced have not been enough to address the problem, and 
backlogs have continued to increase. As depicted in the following chart, in each of the five 
years from 1997/98 to 2001/02, the number of charges disposed of by the Ontario Court 
of Justice did not keep pace with the number of charges received. 
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Ontario Court of Justice 
Five-Year Summary of Criminal Charges Received and Disposed 
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There are serious ramifications when backlogs in courts are not adequately addressed: the 
public can develop a perception that the courts are not responsive to their needs; 
defendants can take advantage of delays to argue that their cases be withdrawn; and 
witnesses’ memories can fade over long periods. Also, long delays caused by backlogs are 
unfair to accused persons who want criminal charges outstanding against them resolved as 
soon as possible. 


The Chief Justices for the Ontario Court of Justice and the Superior Court of Justice 
recently indicated that they have been concerned for some time over the growing backlog 
of cases both in criminal and in family matters, particularly in the larger urban court 


locations. Both of these Chief Justices have informed the Attorney General that additional 
judicial appointments are necessary to reduce backlogs. 


Despite the Ministry’s efforts, the number of backlogged cases in 2002 was at its highest 
level in 10 years. There is a risk that a situation similar to 1992 may be developing, when 
long delays resulted in more than 50,000 charges being withdrawn from prosecution. 


Recommendation 


The Ministry should work with the Judiciary and other stakeholders to develop 
more successful solutions for eliminating backlogs, including: 


* creating better tools to identify the sources and specific reasons for delays 
so that action can be taken to address potential problems in a more timely 
manner; 
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e assessing the resource implications of actions taken and decisions reached 
by the different parties to a trial so that resources allocated to courts can 
handle the increased caseloads; and 


e establishing realistic targets and timetables for eliminating backlogs. 


Ministry Response 


The Court Services Division has undertaken a project to improve the collection 
and analysis of management information to monitor case processing, to 
identify and diagnose processing delays, and to disseminate management 
information to the field, other divisions, the Judiciary, and others, in order to 
support prevention and resolution of delays. 





Options for enhancing the scope of management information available from 
the Division’s data system are being explored. The new case-tracking system 
for civil and family proceedings will provide the Division, for the first time, with 
data about the length of those proceedings. In the short term, the criminal 
operational system will be upgraded to, for example, allow the collection of 
information about the reasons for adjournments. Once preliminary 
improvements have been made to the criminal operational system, options for 
collecting data about the length of criminal proceedings and other valuable 
diagnostic data will be explored. 


The Division is also developing a staff resourcing model and will assist the 
Ontario Court of Justice to develop a judicial complement model to ensure 
appropriate and effective resource allocation. 





The government will invest $15.8 million in new funding this year to ensure that 
the justice system operates more efficiently and more effectively. This 
investment enables the Attorney General to appoint 15 new judges to the 
Ontario Court of Justice and hire a prosecution team, including at least 36 new 
Crown attorneys and other staff needed to support court operations. 


More resources and better information will allow for the establishment of 
realistic targets and timelines for eliminating backlogs. 








INFORMATION SYSTEMS AND THE USE OF NEW 
TECHNOLOGIES 


The Division uses two main computerized systems to provide information to the Judiciary 
and Crown attorneys. The Integrated Court Offences Network (ICON) is an on-line 
mainframe system that accumulates information by courthouse in the Ontario Court of 
Justice. It maintains case data and produces court docket and monthly statistical reports. In 
the Superior Court of Justice, monthly statistical reports are produced by the Court Input 
Statistics System (CISS) through information collected from individual courts using manual 
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or stand-alone computer systems. In addition, several courthouses use a variety of local 
systems to schedule civil cases. 


Integrated Justice Project 


In 1997, we reported that the ICON system, which was established in 1989, was outdated 
and did not capture all the necessary information or produce the reports needed by both 
the Judiciary and the Ministry. There were concerns with these systems over inconsistencies 
in the classification of data, data accuracy, and timeliness of information. 


In 1996, the Ministry along with other justice ministries and a consortium of private-sector 
partners, initiated the Integrated Justice Project (IJP), which was created with the intention 
of facilitating a more modern, effective, and accessible administration of justice. New 
integrated information systems were expected to be developed for police, Crown attorneys, 
courts, and corrections. The courts component of IJP was the largest and most complex. It 
was expected to introduce new or improved systems for case management, court scheduling, 
electronic document filing, and digital audio recording of official court records. 


Because of significant cost increases and delays, however, the IJP project was terminated in 
October 2002. No new systems were delivered to courts, except for a few test projects. The 
Ministry has now assumed responsibility for any new court information systems and, where 
possible, continues any development carried over from the IJP project. 


During the period from 1996/97 to 2002/03, the Ministry invested approximately 

$21 million in IJP and made minimal enhancements to existing court information systems, 
since the expectation was that these systems would be replaced. As a result, both the 
Judiciary and Ministry informed us, although the IJP has had some benefits in improving 
information sharing and communications, on the whole, the project has actually delayed 
any real progress in introducing new information technology to the courts. 


We reported on the IJP in our 2001 Annual Report, which was followed by a number of 
Standing Committee on Public Accounts hearings in 2002. During the hearings, the 
Committee considered the IJP’s financial and qualitative benefits to users of the justice 
system and concluded that the IJP was essential to the modernization of the justice system in 
Ontario. The Committee indicated these benefits might not be realized if the project were 
terminated; therefore, it encouraged the ministries involved to proceed with the IJP either 
under new arrangements with the Consortium or through alternative approaches. 


With the termination of the IJP, we reiterate the concern from our prior audit that the 
courts’ information systems were in critical need of modernization to reduce inefficiencies 
and reduce delays in the administration of justice. 
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New Technologies 


Over the past few years, several new technologies have been introduced to certain courts. 
However, most new technology initiatives were conducted on a test basis and have yet to 
improve the efficiency of the courts. Two of these initiatives were as follows: 


¢ The volume of cases being handled each year requires court staff to manage a large 
number of documents as they are processed through the courts. The savings and 
efficiencies of moving to a paperless, electronic court document system would be 
substantial. Since 1996/97, as part of an IJP test project, the Ministry has accepted 
certain electronic forms filings from lawyers, primarily for civil and small claims court 
documents. ‘To date, however, the use of this service has been relatively low, and only 
three courthouses accept electronic filing. 


In addition, there were few benefits to the Ministry from this initiative because the 
Ministry’s existing systems did not have the capability for processing the electronically 
filed forms. Instead, the forms had to be printed out by court staff and the information 
entered manually into the local systems. 


¢ Court transcripts are recorded manually by court reporters. This often delays court 
proceedings because long periods of time are often taken to produce lengthy court 
transcripts. The IJP tested digital audio recording systems in three locations with the 
intention of moving all courts to an automated court reporting system, both to improve 
the timeliness of obtaining transcripts and to reduce court reporter costs. However, the 
Ministry determined that the software did not meet Ontario’s functional requirements 
for courtroom recordings, nor would it realize the goal of reducing staffing costs. 


On the other hand, one area where the courts made good use of technology was with video 
appearances. Most criminal court appearances are for preliminary or remand hearings, 
which may take a few minutes to complete, and after which the accused is remanded or 
returned to custody to await trial. The Video Remand Project allows an accused person to 
appear in a courtroom by video conferencing from a correctional institution or police 
station. This eliminates the cost and need to transport the prisoner to court. As of March 31, 
2003, approximately 100 courts and detention locations have installed equipment that 


allows for video remand appearances. 





Recommendation 








To help ensure the timely disposition of cases and improve efficiencies, the 
Ministry should take the necessary steps to upgrade the information 
technologies used in courts. In addition, the Ministry should establish a 
comprehensive plan for the timely implementation of new information 
technologies. . 
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| Ministry Response 


The Ministry is committed to modernizing the justice system to increase public 
safety, improve service, and increase access to justice. Any new development 
of system enhancement by the Ministry are to be based on business cases that 
are affordable, staged, and cost effective and that set realistic time frames. As 
part of its five-year plan, the Ministry has committed to a court-case-tracking 
system named “Frank” and other upgrades and plans. 


Frank is a court-case-tracking system for civil, family, small claims, divisional, 
and superior court criminal cases that will operate in all Ontario courthouses. 
It has been piloted at courthouses in Brantford, Oshawa, Sudbury, and Whitby. 
It helps staff ensure that cases are managed within prescribed timelines and 
eliminates time-consuming manual processes. To further improve courthouse 
efficiency, Frank generates regulated notices and orders, allows court staff to 
make case-specific inquiries without having to retrieve paper case files, and 
prevents duplication of efforts with a system that requires data to be entered 
only once. 


In addition, the Ministry has committed to: 


¢ make upgrades to the operational system for criminal proceedings in the 
short term to provide more data to identify opportunities to increase 
efficiency, while, in the longer term, options for an improved criminal-case- 
management system will be developed; 


e develop a multi-year strategic information management/nformation 
technology plan to define the information technology priorities and identify 
resource requirements; and 


¢ continue with the model electronic courtroom, which was introduced to 
pilot the use of multimedia presentations and evidence, remote 
appearances, and digital audio recording and which is operated and 
managed by the Toronto region. 





FINANCIAL INFORMATION 


Financial information is needed to properly assess accountability for expenditures and to 
help determine whether court services are provided economically and efficiently. 


We reported in 1997 that the Ministry made little effort to assess its costs, other than to 
compile information on actual compared with budgeted expenditures by region and court 
location. In 2003, the Ministry still did not have any regular management reporting systems 
in use, at either the corporate or regional level, that would allow management to monitor 
how cost effectively court services were being delivered. For example, none of the regions we 


visited reported their costs by activities or compared their costs with any benchmarks or costs 


in prior years or against costs for similar services at other regions; nor were costs reported in 
standardized reports that would allow for meaningful comparisons. 
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The Ministry provided us with reports produced at the corporate level that contained better 
financial and management information on program costs. For example, a draft report for 
January 2002 was prepared that compared various court activities and costs by region and 
with other provinces. The report was not complete or verified, and the Ministry 
acknowledged the need for additional data. As of March 2003, no final or subsequent 
reports had been produced. 


Recommendation 





To manage the cost of court operations effectively, the Ministry should: 


e identify and collect the information needed to assess whether court 
services are being provided economically and efficiently; and 


e determine how mor een technology can best be utilized to facilitate this 
process. 


Ministry Response 


In its five-year plan, the Ministry has established the following measurable 
service standard: Divisional finances are managed according to Management 
Board standards and policies and are tracked monthly to ensure the [Court 
Services] Division operates within its allocation. 





Effective April 2004, the Division will implement an expenditure account 
structure that will allow for the capture of costs by practice area. A review of 
the Division’s expenditure account structure is currently underway to 
determine how best to implement this new structure. This review will also 
include capturing staffing costs through Corpay coding. 


The implementation of the Integrated Financial Information System (IFIS) is 
planned for fall 2004. The revised coding structure will be incorporated into 
IFIS. IFIS will provide improved expenditure-commitment reporting to assist in 
forecasting. 


A review of regional expenditures is currently Hee GY Particular attention is 
being given to costs related to caseload. 





Court costs will be linked with the improvements planned for the collection of 
workload statistics. It is expected this will allow for the determination of costs 
based on case type. 





EXPENDITURE CONTROLS 


Our audit identified several instances where the Ministry did not comply with the 
applicable Management Board of Cabinet directives that require that goods and services, 
including consultants, be acquired economically and competitively, and that payments be 
properly controlled. For example: 
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¢ The directives set out operating procedures for the use and control of purchasing cards, 
which are credit cards issued to government employees, allowing them to make minor 
purchases more cost effectively. We noted that controls over local purchasing cards were 
often not followed by staff at courthouses and regional offices. About half of the 
monthly statements we examined were not reviewed and approved by the cardholders’ 
manager, and receipts were often missing. 


¢ We noted two occasions where the Ministry hired contractors to make repairs at 
courthouses without signing an agreement that set out the terms and conditions of the 
assignment or the terms for payment. In addition, the Ministry could not produce any 
documentation regarding the competitive selection of these contractors. One contractor 
was paid more than $100,000 from January to March 2002, while the other contractor 
was paid more than $400,000 over the same time period. 


The Ministry also engaged several consultants to perform managerial or operational work 
on an ongoing basis, contrary to the Management Board of Cabinet requirements. Por 
example, the Ministry has hired one consultant since 1989 to provide varying degrees of 
modifications and support to a court office system. Over the last three years, this consultant 
was paid approximately $565 per day for a total of $253,600. This practice is contrary to 
the Management Board of Cabinet directive on consulting services that stipulates that, 
where appropriate, a transfer of knowledge must occur from consultant to staff to avoid a 


continuous reliance on consultants. 












Recommendation 


The Ministry should ensure that adequate controls are in place over 
expenditures so that goods and services, including consultants, are acquired 
competitively and in compliance with Management Board of Cabinet directives. 





Ministry Response 


The Ministry is committed to ensuring that adequate controls are in place over 
expenditures. 


Communications pertaining to proper procurement procedures and payment 
processing will be sent to all Division management for distribution as 
appropriate. Links to Shared Services Bureau sites will be made available to 
managers. 


The topic of adequate controls over expenditures will be addressed on a 
periodic basis at managers’ meetings to reinforce adherence to proper 
practices. 
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CAPITAL PROJECTS 


Over the past six years, the Ministry spent approximately $275 million on capital projects to 
modernize and improve courts. This included renovations to existing courthouses or 
consolidation of several local court locations into a new courthouse. A Management Board 
of Cabinet directive requires the Ministry to arrange for construction and management of 


capital projects by the Ontario Realty Corporation (ORC), or its private company agent. 


Our audit of the acquisition and management of capital projects for courts identified the 
following deficiencies: 


Leasehold improvements at new courthouse—2201 Finch Avenue West, Toronto. As of 
March 2003, construction was substantially completed at the Ministry’s new 73,000 f° 
courthouse at a leased accommodation at 2201 Finch Avenue West. Total costs to complete 
the project were expected to be $30 million. The new court location replaced an existing 
leased courthouse that was discovered in November 2000 to contain toxic mould and 
therefore to present health concerns for its occupants. 


In May 2001, the Ministry initially leased approximately 40,000 ft’ at the new location as 
an interim courthouse, while remediation and refurbishing were expected to take place at 
the existing leased courthouse. We were informed that the project at the interim site was 
managed by ORC’s agent on an expedited basis, with expected occupancy by September 1, 
2001. Therefore, the lease for the interim site was not selected through a call for tenders. 
Rather, we were advised that ORC considered several locations using an independent realty 


broker. 


The renovation project to convert the interim location to a temporary courthouse was not 
awarded using a call for tenders either. Instead, the Ministry informed us that the ORC’s 
agent managed the project using a contractor selected from a list of unit-priced suppliers 
who provided their services at rates that were agreed to beforehand. While the rates quoted 
by these suppliers were obtained through a pre-qualification process, the suppliers on the list 
were meant to be used on assignments costing less than $100,000. The project, then 
estimated to cost approximately $8 million, was therefore carried out without a fixed-price 
contract and without a proper competitive acquisition process for a project of this size. In 
addition, the Ministry did not obtain ministerial or Management Board of Cabinet 
approval for not following competitive selection procedures before awarding a contract of 


this magnitude. 


In September 2001, the Ministry sought and obtained approval from the Management 
Board of Cabinet and the Cabinet Committee on Privatization and SuperBuild (CCOPS) 
to change its plans and establish a new courthouse at the Finch Avenue leased location. The 
Ministry stated that the change was needed after a more detailed building condition 
assessment found extensive renovations were necessary at the existing courthouse. The 
revised total cost of renovations for the new courthouse at the Finch Avenue location was 
then estimated at $21 million, with additional space requirements of 33,000 ft. In 
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addition, the expected occupancy date was changed from September 2001 to January 
2002 for court operations, and to May 2002 for the construction of holding cells. 


By March 2003, the cost of the project had increased to $30 million, about $9 million 
more than forecast in September 2001. Only $6 million of the total cost of the project was 
awarded competitively. In addition, we noted that other courthouse projects had 
significantly lower construction costs. Costs for these other projects ranged from 
approximately $200 to $280 per ft’ (in 2001 dollars) when using contractors selected 
through calls for competitive tenders and fixed-price contracts. The Finch Avenue project 
incurred costs of approximately $390 per ft’, which were 40% higher than the costs for 
other similar projects. 


The Ministry relocated all of its operations out of the existing leased courthouse as of July 
29, 2002, or six months after the expected occupancy date, with the holding cells not 
completed until March 2003, requiring other courthouses to be used for an extended 
period. 


In addition, the Ministry informed the Management Board of Cabinet and CCOPS when 
seeking approval for its new courthouse that the primary cause of the toxic mould problem 
was the inadequate building maintenance provided by the former building owners over 
many years. However, according to the Ministry and ORC, after obtaining legal advice, a 
decision was made to continue to pay rent on the vacated courthouse until the lease expires 
on April 30, 2004. This rent will total more than $1 million from the time the building was 


vacated. 


Notwithstanding the Ministry’s and ORC’s efforts to deal as quickly as possible with the 
health concerns at the former courthouse, during the period of more than two years taken 
to complete the project, large contracts were awarded without following competitive 
selection procedures, and approvals to deviate from required Management Board of 
Cabinet directives were not obtained from either ministry senior management or Cabinet. 


Mould remediation work—Newmarket courthouse. In March 2000, toxic mould was 
discovered in the Newmarket courthouse. The building was closed on June 30, 2000, 
because of escalating concerns for the occupants’ health and to allow for the necessary 
remedial repairs and related updates to the building’s systems. The courthouse was 
reopened in July 2001, after approximately $23 million had been spent on remedial work. 
The Ministry also spent an additional $20 million for temporary accommodations to 
maintain court operations during the period. 


The costs and scope of the remedial work were significantly underestimated throughout the 
life of this project. On April 20, 2000, the total project costs were originally estimated at 
approximately $250,000, and there was no expectation that court operations would need to 
be temporarily relocated during the repairs. One month later, when substantially more 
serious mould damage than had previously been determined was found to exist, ORC’s 
agent estimated the total costs of the remedial work, in a “worst-case scenario,” to be 

$3.5 million. However, the estimated costs were revised upwards as follows: $8.3 million (in 
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August 2000); $11.4 million (in October 2000); $14.9 million (in December 2000); and 
$18.4 million (in March 2001). As noted earlier, the final costs for the remedial work were 
approximately $23 million. 


We noted that the majority of contracts for both the remedial work and the relocation to 
temporary accommodations were awarded without following competitive selection 
procedures. For example, on May 12, 2000, as part of the original project, a contractor was 
selected through a competitive process to do remedial work at a cost of $52,000. In spite of 
the increase in the scope and extent of the work and the significantly higher estimates of 
project costs that followed, no competitive quotes were subsequently obtained from other 
contractors. Eventually, the primary contractor received $23.8 million, of which 

$13.4 million was for the mould remediation work. The contractor was also awarded 
contracts without competition, totalling $10.4 million, for work pertaining to the 
construction of temporary accomodations. 


When the building was closed on June 30, 2000, the remediation project was expedited to 
minimize the time required for the Ministry to maintain court operations from temporary 
accommodations. However, the significant difference between the original estimate and 
final project costs demonstrates that, even when projects are expedited, adequate upfront 
planning ts imperative. Such planning would allow for better and more predictable 
decision-making prior to commencing capital projects and during the entire life of the 
projects and would ensure that appropriate competitive procurement practices are 


followed. 


New victim witness facilities and Crown attorney offices—Milton courthouse. As noted 
above, a directive from the Management Board of Cabinet requires the Ministry to use the 
ORC for construction of capital projects. Contrary to this directive and without seeking 
competitive tenders, in February 2001 the Ministry contracted directly with a private 
project management company to construct victim witness facilities and Crown attorney 
offices at the Milton courthouse. The Branch expected the project management company 
to hire contractors on a competitive basis to perform the construction. The project was 
estimated to cost $200,000. 


More than one year later, the project management company provided the Ministry with an 
invoice indicating the work had been completed, even though no contractors had been 
hired and no work had been started on this project at the time. In March 2002, the 
Ministry paid the company’s invoice for $187,000, even though it was aware no work had 
been done. We were informed that the payment was made because the funds were 
committed for that fiscal year and may not be available at a later date. 


Following our inquiries into this project, the Ministry terminated its contract with the 
project management company in January 2003. The Ministry then re-assigned 
management of the project to the ORC. At that time, the project management company 
still had not completed any substantial work on the project. In January 2003, the Ministry 
requested and received a refund from the project management company of $166,000. 
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In summary, better planning, better project management, the use of competitive tenders, 


and compliance with corporate polices could have resulted in lower costs to complete capital 


projects at courthouses. 





_ Recommendation 


| To ensure that courthouse construction and renovation projects are acquired 
competitively, on budget, and in accordance with Management Board of 
Cabinet policies, the Ministry, in conjunction with the Ontario Realty 

_ Corporation, should adequately plan and manage its capital projects. In 

_ addition, the Ministry should ensure that appropriate controls are in place so 
that contractors are only paid for completed work. 


Ministry Response 


Mould presents a number of possible health risks. The Health Promotion and 

_ Protection Act requires a building manager to provide a safe environment for 
| occupants of the building. The Occupational Health and Safety Act details the 
_ responsibility of the employer to ensure a healthy and safe workplace for its 

_ employees. 


_ At the courthouse at the previous location (80 The East Mall), which is the 

_ fourth-largest courthouse in Ontario and which deals with criminal and young 
offender violations, the Ministry of Labour issued six orders (from late fall 2000 
to summer 2001) relating to serious mould, structural, heating, ventilating, and 

| air-conditioning issues. To address these issues, some court operations were 
transferred to different locations on an emergency basis until the new 

_ courthouse at 2201 Finch Avenue West could be completed. 


In Newmarket, 27 Ministry of Labour orders were issued during the spring of 
_ 2000 and resulted in work refusals. The Ministry undertook the difficult 

| decision to shut down the courthouse to remediate the mould. A temporary 
portable facility was employed at the site, and some court operations were 
moved to nearby locations to ensure access to justice services continued 
without interruption. 


In both cases, the nature of the mould constituted an emergency, which the 
Ministry was obligated to address in the most expedient manner possible. To 
help ensure that the Ministry was receiving value for money, an independent 
cost consultant and quantity surveyor was engaged by the Ontario Realty 
Corporation and was responsible for certifying the accuracy and the validity of 
all payments made. Without the emergency measures being taken, there would 
have been a breakdown in the administration of justice and criminal cases may 
have been lost. The actions of the Ministry were in the best interest of the 
public and the staff who work in the two courthouses. 


The Ministry does acknowledge the benefits of economy of costs related to the 
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service providers to ensure construction and renovation projects are acquired 
competitively, on budget, and in accordance with Management Board of 
Cabinet policies. 


As part of the capital planning cycle, the Ministry will continue to consult with 
the Ontario Realty Corporation as its mandatory service provider to ensure 
proper planning of capital projects is conducted. 





The Ministry has instituted additional processes and procedures to further 
strengthen the controls in place on its capital allocation. 


COURT SECURITY 


Under the Police Services Act, local police services boards are responsible for ensuring the 
security of judges and persons taking part in or attending court proceedings. The local 
police services contribute to court security primarily by providing and paying for trained 
officers to manage and implement security measures and to operate security devices. The 
Ministry has the responsibility for court security costs that are not related to staffing. These 
include ensuring that courthouses are designed and maintained in an appropriate manner, 
such as having secured corridors and holding cells. Other costs are for purchasing and 
maintaining security devices, such as scanners at entrances and security cameras. 


In March 2001, the Division initiated a court security project. An internal committee 
consisting of senior management was given the mandate to address facility-related security 
risks and to help in the decisions needed to prioritize the short- and long-term capital 
expenditures for court security. As part of the project, the committee engaged an outside 
security consulting company at a cost of approximately $148,000 to review the security risks 
at 90 larger court sites. 


Based on the consultant’s findings, in January 2003 the committee reported numerous gaps 
in security measures at courthouses surveyed. More than 150 types of security deficiencies _ 
were noted. For example, some courthouses had: 


¢ no monitoring of panic buttons by local police or court staff; 

¢ no secure cabinets in judges’ chambers; 

¢ no guards at the main entrances of court facilities; 

¢ no parking for judges that was segregated from the public; and 
¢ no security inspections of courtrooms before proceedings. 


The key risks identified in the committee’s report included: unauthorized weapons; verbal 
abuse and threats; vandalism and sabotage; theft; and assault and altercations. 


During our visits to courthouses, we most often noted significant inconsistencies in the level 
of security at the public entrances to courthouses dealing with criminal cases. While police 


46 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





at many of these courthouses required people entering to be scanned for metal devices and 
to have their baggage checked, police at other courthouses did not impose these 
requirements. At two courthouses we visited, the local police were not operating metal 
scanning devices at entrances even though the Ministry had provided the equipment at a 
cost of approximately $15,000 per courthouse. At another courthouse we visited, where 
these checks were conducted, the local police indicated that their searches resulted in more 
than 100 persons being charged annually for various violations. 


Two of the Chief Justices expressed concerns to us about the level of security in courthouses. 
One of them wrote: 


The issue of security is a significant concern to the court.... While local security 
committees may be in place, there exists across the province a patchwork of security 
measures, largely dictated either by the occurrence of a significant event leading to 
enhanced levels of security at a particular court location or, more commonly, by budget 
constraints leading to reduced security. The present state of security with respect to 
courthouses in the province of Ontario is such that the public and court users may be 
exposed to unnecessary security risks because of the lack of a consistent approach to 
security issues.... 


It is the Court’s position that what is required to improve courthouse and courtroom 
security is a province-wide review of existing security measures combined with the 
creation of specific security standards for courthouses and courtrooms. 


At the time of our audit, the Ministry was in the process of considering how to address the 
security issues that had been raised. 


Recommendation 


To ensure the safety of judges and persons involved in court proceedings, the 
Ministry should act quickly in co-operation with stakeholders to establish and 
maintain an appropriate level of security in all courthouses. 


Ministry Response 


The Court Services Division, in co-operation with stakeholders, is considering 
the steps necessary to ensure the availability and consistent application of 
security devices at all courthouses across the province. 


Building on the court-security-project report findings, the Ministry’s Facilities 
Branch has been requested to undertake efforts towards determination of 
facility standards, including standards for security equipment, for 
consideration in various types of court facilities (for example, older versus 
newer facilities and large versus medium versus small sites). 


COLLECTION OF FINES 


As part of the Local Services Realignment initiative, in January 1997 the Government of 
Ontario decided to transfer to municipalities the administration and prosecution functions 
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for most charges that fall under the Provincial Offences Act, including the collection of fines 
for these charges. Most of the fines transferred were for offences committed under the 
Highway Traffic Act. This transfer of responsibility to municipalities commenced in March 
1999 and concluded in February 2002. Afterwards, the Ministry only retained 
responsibility for collecting fines for violations under other provincial statutes and the 
Criminal Code. As of February 2003, the Ministry had the responsibility for collecting 
approximately $60 million in outstanding fines, about half of which were at least five years 
old and considered by the Ministry to be uncollectible. 


The Integrated Court Offences Network (ICON) is used to keep track of the fines, and it 
produces a monthly report on the amounts imposed and collected and on the age of 
outstanding fines. Approximately $15 million in fines are imposed annually, and the 
majority of these fines require collection efforts. The Ministry has an agreement with 
Management Board Secretariat’s Collection Management Unit (Unit), which requires that 
outstanding fines be transferred to the Unit at least every three months for collection by 
private collection agencies. Timely transfer to collection agencies is imperative because the 
longer a fine is outstanding, the more difficult it is to ultimately collect. 


In 1997, we reported that the Ministry needed to improve its efforts to collect fines. 
However, we found there had been little progress made in implementing our 
recommendations. Specifically: 


¢ Over the last five years, the Ministry made transfers to the Unit on only four occasions. 
The last transfer was made on October 1, 2002, which was two-and-a-half years since 
the previous transfer. Also, the Ministry was not reconciling its records to those kept by 
the Unit to ensure that all fines in default had been transferred and accounted for. As of 
February 28, 2003, the Unit reported only $33 million in fines outstanding for longer 
than three months. We noted that at the same time the Ministry’s records showed 
approximately $57 million in fines that were in arrears for longer than three months. 


¢ The Ministry receives monthly activity reports from the Unit on fines collected and 
outstanding. However, the Ministry did not monitor collection rates to determine the 
success of collection efforts. 


¢ According to its agreement with the Unit, the Ministry is required to authorize the 
various types of enforcement measures to be used by private collection agencies. We 
noted that the enforcement measures authorized by the Ministry were among the 
weakest in use when compared with those utilized by other ministries. For example, the 
Ministry does not permit: reporting defaulters to a credit bureau; accumulating interest 
and collection charges; suspending a driver’s licence or denying vehicle plate renewal; 
seizing income tax refunds; garnishing wages; or taking legal action. 


Enforcing the collection of fines is necessary to ensure the integrity of the justice system and 
to deter offenders from re-offending in the future. 
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Recommendation 


To better ensure that offenders pay their fines, the Ministry should: 
forward all outstanding fines to the Collection Management Unit for 
collection on a timely basis; 


authorize more vigorous enforcement measures to pursue outstanding 
fines; and 


improve its system for tracking fines. 


Ministry Response 


Of the total fines imposed annually, historically about two-thirds are eventually 
collected. The remaining one-third of the fines is difficult to collect. 


To ensure that offenders pay their fines, the Ministry will: 


ensure regular (quarterly) transmission of defaulted fines data to the 
Collection Management Unit (Unit); 


arrange for a dedicated resource to liaise with the Unit on collection activity 
and develop a funding proposal for the 2004/05 fiscal year; and 


investigate and implement improved tracking of fines and more vigorous 
enforcement measures. 2 


PERFORMANCE REPORTING 


Good performance reporting should include the following attributes: clear goals and 
objectives; complete and relevant performance measures; appropriate standards and targets 
for measuring results; reliable systems to gather the necessary information; and a reporting 
mechanism for regularly communicating accomplishments and areas requiring corrective 
action. Because responsibility for the courts is shared between the Court Services Division 
and the Judiciary, both parties have to participate in establishing appropriate performance 
measures. 


In 1997, the Ministry informed us that it was in the process of developing performance 
measures for the administration of courts, including standards and targets, for inclusion in 
the Ministry’s business plans. The Ministry intended to develop more specific performance 
indicators, targets, and benchmarks against which achievement of results could be 
measured. In our 1999 follow-up, we noted that these plans were still in progress. Our 
current audit found that the Ministry had still not made any significant improvements to 
measure and report on its performance. 


As at March 31, 2003, the Ministry had only two performance measures in its annual 
business plan on courts. These consisted of the results of surveys of public satisfaction with 
services in small claims court and in family law information centres, and a record of the 


Court Services 49 








| VFM Section 3.01 | 


-VEM Section 3.01 








percentage of civil cases settled through mediation. These areas represent only a small 
fraction of the services provided by courts. No outcome or activity measures were available 
on the core businesses of criminal courts and judicial services. In addition, we noted that the 
Ministry had not established any indicators to measure and report on efficiency, such as the 
costs of providing court services or a comparison of Ontario’s costs with those of other 
jurisdictions. 


We noted that the Ministry had some information that was available internally to its 
management. This information consisted of data on backlogs, the number of court sitting 
hours, and the average time to trial in the Ontario Court of Justice. The information could 
have been reported on publicly in the Ministry’s business plan or Web sites. Other 
jurisdictions also have relevant indicators that they reported, such as court workloads, 
administrative duties completed within targeted time, the collection of fines, and waiting 


times for trials. This information was already available within the Ministry but had not been 
published. 


The Ministry acknowledged the need for improvements in performance information that 
could help establish service standards and that could support its business goals and resource 
needs in an environment of limited financial resources. The Ministry indicated that the 
necessary financial and management information would be gathered as part of its recent 
efforts to establish a five-year strategic plan. 





Recommendation 


The Ministry should measure and report on its cost-effectiveness, efficiency, 
and outcomes in providing court services by: 


¢ working with the Judiciary to develop appropriate performance indicators 
and targets against which it can measure the achievement of its business 
goals and operational standards; 

* ensuring its information systems gather and report the information needed 
for management to monitor performance on an ongoing basis; and 


e reporting regularly to the public on its performance. 





Ministry Response 


A five-year plan has now been developed for the Court Services Division 
identifying five business goals and 42 measurable service standards for 
compliance with business goals, including both overarching standards for the 
Division and standards for each practice area. The goals, standards, and 
initiatives established in the five-year plan will be reflected in the Division’s 
Business Plan and will be published in the Division’s new Annual Report this 
summer. All aspects of the plan, including the service stancards, will be 
reviewed and updated annually in consultation with the Judiciary and court 
users. As a part of this process, measures to gauge the Division’s cost- 
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effectiveness, efficiency, and outcomes in providing court services will be 
explored. 


Performance measuring will also be supported through the evaluation of all 
Division programs on a four-year cycle. The evaluations will determine the 
relevance of the program to the core business of the Ministry and assess the 
efficiency, effectiveness, and sustainability of the program. Program 
evaluations for enforcement activity and judicial support services will be 
completed this December, for criminal and head-office operations in 2004/05, 
and for civil and family operations in 2005/06. 


The Division has a number of initiatives underway to provide enhanced 
management information to support management decision-making and 
performance reporting. These measures will improve the scope and accuracy 
of, and access to, management information. For example: 


e Anew case-tracking system named “Frank,” for civil, family, Small Claims, 
Divisional Court, and Superior Court of Justice criminal proceedings, will 
provide more, and more accurate, data about those proceedings. 

A project plan has now been developed to improve the quality of data 
collection in the Integrated Court Offences Network and to expand the 
scope of data available. 





a 


Court Services 5 


VEM Section 3.01 


| VEM Section 3.02 








MINISTRY OF COMMUNITY, FAMILY 
AND CHILDREN’S SERVICES 


3.02—Children’s Mental 
Health Services 





BACKGROUND 


The Child and Family Intervention (CFI) and Child Treatment (CT) programs of the 
Ministry of Community, Family and Children’s Services are collectively referred to as 
“Children’s Mental Health Services.” These programs fund transfer-payment agencies that 
provide services to children and/or the families of children who have social, emotional, or 
behavioural problems or psychiatric disorders. Services are also offered in cases such as 
family breakdown, physical or sexual abuse, attempted suicide, and depression. The CT 
program generally deals with more severe cases of children who are treated by or under the 
supervision of a psychiatrist; otherwise, the CFI and CT programs are similar. 


Under provisions of the Child and Family Services Act, approximately 250 community-based 
agencies are funded. The types of services offered include assessment, psychiatric therapy, 
counselling, crisis intervention, and skills training and education, as well as residential-based 
services (mental health services offered in a residential setting) to children who require more 
intensive assistance. The Ministry’s nine regional offices are responsible for contracting for 
and monitoring the delivery of children’s mental health services by the agencies in their 
respective regions. However, since children’s mental health services are not mandatory, 


services provided are limited by the level of available funding rather than the level of need. 


Services are generally provided to children under the age of 18, including young offenders 
and children who may already be receiving services from a Children’s Aid Society or other 
programs funded by the Ministry. 


Total Children’s Mental Health Services expenditures have increased substantially since the 
time of our last audit, from $213 million in 1997 to $315 million in 2002/03. However, 
most of this increase has been spent on several new initiatives in the last two years that in 
most cases provide intensive services to relatively few individuals with complex special needs. 


The following chart shows how funding was distributed among components of Children’s 
Mental Health Services. 
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Children's Mental Health Services Expenditures, 
2002/03 Fiscal Year 
($ million) 


Non-residential 
services 
$153.2 (49%) 


Residential-based 
services 
$89.8 (29%) 
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AUDIT OBJECTIVES AND SCOPE 


Our audit objectives were to assess whether the Ministry’s administrative procedures were 
adequate to ensure that: 


¢ the quality and outcome of services provided by the community-based agencies was 
monitored and assessed; and 


¢ transfer payments to agencies were reasonable and satisfactorily controlled. 


The scope of our audit included a review and analyses of ministry files and administrative 
procedures, as well as interviews with appropriate staff at the Ministry's head office and three 
regional offices. We also visited a number of agency sites and interviewed staff there. 


Prior to the commencement of our audit work, we identified the audit criteria that we 
would use to conclude on our audit objectives. These were reviewed with and agreed to by 
senior management of the Ministry. 


Our audit work was primarily conducted in the period from November 2002 to March 
2003. Our work on program expenditures and procedures focused on the 2001/02 and 
2002/03 fiscal years. Our audit was performed in accordance with the standards for 
assurance engagements, encompassing value for money and compliance, established by the 
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Canadian Institute of Chartered Accountants, and accordingly included such tests and 
other procedures as we considered necessary in the circumstances. 


We did not rely on the Ministry’s internal audit branch to reduce the extent of our audit 
work because they had not recently conducted any work in the area of our audit. 


OVERALL AUDIT CONCLUSIONS 


Based on the audit work we performed, we concluded that the Ministry was not adequately 
monitoring and assessing the quality of the services provided by agencies. As a result, the 
Ministry cannot be assured that vulnerable children in need are receiving the care and 
assistance they require. In particular, the Ministry: 


¢ had not established service quality standards and service evaluation criteria to help staff 
monitor whether or not services were of an acceptable quality and represented value for 
money spent; 


¢ had not established waiting-time standards for access to service that were reasonable and 
commensurate with individual children’s needs, and was not monitoring the extent and 
impact of lengthy waiting times for service; and 


* was not receiving or assessing information from agencies about the outcomes of the 
services they were providing, and could not take necessary corrective action with respect 
to financial and operational results due to the ineffectiveness of the existing quarterly 
reporting process. 


We also concluded that the Ministry's administrative procedures were not adequate to 
ensure that transfer payments to agencies were reasonable and satisfactorily controlled. 


Specifically: 


¢ The Ministry’s funding decisions were not based on sufficiently detailed and relevant 
financial and operational information from agencies to ensure that the amounts 
approved were commensurate with the demand for, and value of, the services to be 


provided. 


¢ The Annual Program Expenditure Reconciliations and accompanying audited financial 
statements that the Ministry received did not contain sufficiently detailed and 
comparable information to allow it to detect ineligible and inappropriate expenditures 
and to determine whether there were funding surpluses. 


¢ The Ministry was not effectively recovering annual funding surpluses from agencies as 
required by Management Board of Cabinet directives. 


¢ The Ministry did not ensure that its management information system provided 
sufficiently detailed, relevant, and accurate information to allow it to monitor the cost 
effectiveness of service delivery. 
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Our conclusions and findings were of particular concern because many dealt with issues we 
had previously raised in our 1997 audit of the CFI program. Although the Ministry agreed 
with the recommendations in that audit and agreed to implement the necessary corrective 
action, its progress has been less than satisfactory. The current status of our 1997 
recommendations ts as follows. 


Current Status of 1997 Recommendations 









1997 Recommendation _ Current Status 


Service quality standards should be established, and service _ | Not fully implemented— 
quality should be periodically evaluated. work in progress 
Outcome indicators should be implemented to assess and Not fully implemented— 
improve program effectiveness. work in progress 

Agency funding decisions should be based on sufficiently Not implemented 

detailed and relevant information from agencies. 



















Where the costs of similar programs are not comparable, the Not implemented 


reasons should be explained and justified. 








Annual Program Expenditure Reconciliations and audited Not implemented 
financial statements should provide sufficiently detailed 
information to permit the identification of inappropriate and 


ineligible expenditures. 


More effective procedures should be in place to recover Improvement noted 
funding surpluses. 


Information necessary to determine whether services are Not fully implemented— 
provided cost effectively should be obtained. work in progress 




















Overall Ministry Response 

















Under the Child and Family Services Act, the Ministry of Community, Family 
and Children’s Services has the legislated authority to provide and/or fund 
services to children who have social, emotional, behavioural, and/or psychiatric 
problems. However, these services are discretionary under the Act and are 
provided to the level of available resources. 


The Ministry is building on its recent experience and is developing a policy 
framework for children’s mental health that will: confirm the government's 
expectations of children’s mental health service providers; identify core 
children’s mental health services; and establish a common policy platform for 
both the children’s mental health services funded by the Ministry of 
Community, Family and Children’s Services and those funded by the Ministry 
of Health and Long-Term Care. The Ministry expects that the policy framework 
will help clarify roles, responsibilities, and accountability requirements and 
facilitate the delivery of children’s mental health services that focus services 
on children most in need. 
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DETAILED AUDIT OBSERVATIONS 
MONITORING OF SERVICES PROVIDED 


Quality of Service 


Measurable and meaningful quality-of-service standards are essential for ensuring that 
service recipients’ needs are adequately met and that the services provided represent value 
for money spent. Such standards would include set expectations for staff qualifications and 
staff-to-client ratios. Having appropriate quality-of-service standards is particularly 
important in light of the fact that the Ministry is not currently able to monitor and assess 
service outcomes (as discussed further under “Performance Measurement”). 


However, as was the case in our last audit, we found that, for the vast majority of the 
programs funded, neither standards defining acceptable service nor criteria for evaluating 
service quality had been developed. As a result, the Ministry did not have adequate 
assurance that the programs it funded adequately met client needs or represented value for 
money spent. 


For example, in that regard we noted that in July 2001, a regional office that we visited and 
a transfer-payment agency jointly hired a consultant to perform a clinical review of the 
agency's operations. The consultant examined the quality of services offered at the agency 
and found the following: 


* In the absence of established standards, the agency fell short of achieving the standard of 


practice that the consultant expected. 


¢ There were concerns about the strength and consistency of the clinical programs 


offered. 


¢ There were concerns in general about waiting times for service. For one of the agency’s 
programs in particular, the waiting time to serve a client who might be suicidal or violent 


could be months. 


On the other hand, we are pleased to note that the Ministry had developed service 
guidelines for several of the newly established programs. These program guidelines outline 
ministry requirements for core program design and best practices and in some cases include 
specific guidance for staff qualifications and experience, caseload sizes, and intensity of 
intervention. However, the Ministry had not taken full advantage of this initiative because, 
although head office staff considered compliance with the guidelines to be mandatory, staff 
in regional offices in most cases did not, and they therefore did not monitor whether or not 
guidelines were adhered to. 
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Recommendation 
To ensure that agencies are aware of the Ministry’s service-delivery 


expectations and to assist ministry staff in assessing whether services are of 
an acceptable quality and represent value for money spent, the Ministry 
should: 


e establish standards for acceptable service quality, as well as criteria for 
evaluating service quality, for all Children’s Mental Health Services 
programs that it funds; and 


¢ periodically evaluate the quality of services provided and work with its 
partner agencies to take corrective action where necessary. 


Ministry Response 


For the past year, the Ministry has been working with a group of internal and 
external stakeholders in children’s mental health to define core services and 
articulate the Ministry’s vision of service for this sector. The Ministry will 
complete this work in 2003/04 and proposes to undertake an inventory of 
services provided by children’s mental health agencies. The inventory would 
enable the Ministry to provide children’s mental health agencies with clear and 
consistent direction on the core business and services they are expected to 
provide. To follow up on this work, in 2005/06 the Ministry proposes to conduct 
a program evaluation to confirm that the policy framework is being 
consistently applied by agencies, that core services are being delivered, and 
that children’s mental health services are effective, efficient, and affordable. 





As noted by the Provincial Auditor, the Ministry has required that children’s 
mental health agencies implement standardized intake and assessment tools 
(the Brief Child and Family Phone Interview and the Child and Family 
Assessment Scale). Although these tools are still being refined, they document 
improvements and changes in individual children as a result of services 
provided and received. 





The Ministry recognizes the need to monitor quality of service and has 
undertaken a number of initiatives that focus on improving services. For 
example: 


e In 2003/04, the Ministry will pilot a mental health assessment tool for 
children under age six. 

e The Ministry has identified common assessment tools for children with 
autism that are currently being implemented. 

e In 2004/05, the Ministry will identify methods to measure child 
improvements as part of the Autism Strategy announced in November 
2002. 
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Waiting Lists 

‘Timely access to children’s mental health services is often critical for ensuring the best 
possible outcomes for those children in need of the services. Therefore, the Ministry needs 
to ensure that there are standards for access to service that are reasonable and 
commensurate with an individual’s assessed needs. The Ministry also needs to ensure that 
agencies adhere to the standards. 


However, we found that standards for access to services have not been established. 
Furthermore, with the exception of the autism program, information about waiting lists and 
times was not normally provided to the Ministry (in one region, information about waiting 
lists and times for some agencies providing non-residential services was provided to the 
Ministry during the time of our audit, but the Ministry had not yet reviewed and assessed 
rhe 

Our own review of what information was available indicated that waiting times were often 


lengthy. For example: 


¢ Waiting times for children in the autism program averaged approximately one year for 
initial assessment and another year for actual services. Since the autism program is 
provided to children only up to the age of six, some children who turned six while 
waiting for service were removed from the waiting list without ever having received 
services. As of December 2002, 1,105 children were on the province-wide waiting list, 
compared to 453 children who were receiving the service. 


¢ In one region that we visited, 224 children who had been approved as requiring 
residential-based care were on a waiting list for such care for an average period of eight 
months. Undoubtedly, the fact that all 91 beds available in this region were filled 
contributed to these lengthy waiting times. 


¢ Similarly, in another region that had a consolidated waiting list for non-residential 
services at seven agencies, 138 children were waiting between three and 11 months for 
service. The seven agencies were serving only 28 children in total at that time. 


Unless the regional offices receive reliable waiting-list information from the agencies and 
assess it, the Ministry cannot respond to significant problems on a timely basis. 






Recommendation 


In order that the necessary services are provided to children most in need ona 
timely basis and, when they are not, that the negative impact on children is 
lessened, the Ministry should: 






¢ establish standards for access to service that are reasonable and 
commensurate with individual children’s needs; and 
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e assess the extent to which the standards are complied with and develop 
strategies to monitor and remedy situations where waiting times for service 
are too lengthy. 


Ministry Response 


The Ministry acknowledges that waiting times for children’s mental health 
services may be lengthy. Children’s mental health services are discretionary 
under the Act and are provided to the level of available resources. It is the 
responsibility of service providers to prioritize an individual’s need for service, 
considering ministry guidelines for children “most in need.” 


The Ministry implemented standardized common intake and assessment tools 
and access mechanisms across the province to improve access to services for 
children. However, the children’s mental health sector is currently experiencing 
capacity issues, and the Ministry is working with agencies to prioritize services 
at the community level so that necessary services are provided to children 
most in need. Although the Ministry has provided new funding for children’s 
mental health services since 1999, these funds have been designated to 
address specific service areas and gaps—for example, services to preschool- 
aged children and intensive child and family services—rather than to address 
capacity issues (for example, staffing resources), which would result in 
reduced wait times. 





In 2001 and 2002, the Ministry asked service providers to provide specific 
information on numbers of children waiting for autism services. The Ministry 
used this information to identify additional funding required to meet the need. 
With the announcement of the Autism Strategy in November 2002, the Ministry 
will be spending approximately $100 million on services for children with 
autism by 2006/07. Based on waiting-list information provided, the Ministry 
doubled its budget for Intensive Behavioural Intervention services to respond 
to the demand for service. 








Performance Measurement 
The objective of the Children’s Mental Health Services programs is to provide for a range of 


services that effectively alleviate many types of social, emotional, behavioural, and/or 
psychiatric problems experienced by children and their families. In order to ensure that this 
objective is met, the Ministry must monitor and assess the performance of its transfer- 
payment agencies with respect to both program/client outcomes (changes in clients’ 
conditions brought about by services provided) and program outputs (the types and 
amounts of both services provided and clients served). Such monitoring would enable the 
Ministry to determine whether its expectations were met and, where necessary, to take 
corrective action. 





Children’s Mental Health Services 59 





VFM Section 3.02 | 





| VFM Section 3.02 | 





However, the Ministry still does not effectively monitor performance against measurable 
and meaningful performance targets, for either program/client outcomes or program 
outputs, despite similar findings having been identified in both our 1993 and 1997 audits 
of the CFI program. As noted in our 1993 audit, the Ministry had identified the need to 
improve transfer-payment-agency accountability by measuring agency effectiveness. Again 
in 1997, in response to one of our recommendations, the Ministry agreed that “it needs to 
establish measurable performance targets and indicators and monitor the results achieved 
against the targets established.” 


Although individual agency performance is still not currently being effectively measured 
against pre-established targets, some progress has been made. For example, with respect to 
program/client outcomes, we understand that the Ministry has selected two tools that are in 
the process of being implemented in some agencies: 


* an intake assessment tool for determining the type and severity of a child’s need at the 
time of intake; and 


¢ a performance measurement tool that can be used in certain situations to assess a child’s 
current condition and the child’s improvement over time. 


In establishing a child’s initial condition and measuring changes in the child’s condition 
while being treated, these tools could provide valuable information about program 
effectiveness. However, it has not yet been decided whether any of the information provided 
by these tools will be provided to the Ministry by the agencies and therefore whether the 


Ministry will use the information to assess the outcome of care being provided. 


The Ministry also requests quarterly reports from its agencies. The intent of these reports is 
to compare actual expenditures to those budgeted for and provide planned and actual 
output information such as the number of individuals served and days of residential-based 
care provided. The Ministry generally requires that agencies highlight and explain 
unexpected variances between planned and actual results that exceed 10 percent and 
suggest appropriate remedial actions. These suggestions are to be reviewed and approved by 


the Ministry’s regional staff. 


However, we found this quarterly reporting process to be ineffective for the following 
reasons: 


¢ Output information about the number of individuals served was often unreliable and 
misleading and did not appropriately measure the type and amount of services 


provided. 


¢ Agencies often did not provide the required explanations for, or suggest remedial actions 
for, unexpected variances between actual and planned results. 


¢ There was often no evidence that the Ministry reviewed the quarterly reports—which 
means it also did not review and approve the suggested remedial actions. 
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Recommendation 


In order that children who are receiving mental health services are provided 
with the care and assistance they require, the Ministry should: 


e regularly obtain and assess information about the level and outcomes of 
the services provided by its community-based service-delivery agencies; 
and 


e take the necessary steps to ensure that the existing quarterly reporting 
process is effective in providing reliable and useful information on both 
expenditures and service outputs. 


Ministry Response 


The Ministry agrees that performance measures are an essential part of the 
business planning process and provide important information to support 
funding and program management decisions. 


The Ministry has identified the need to enhance its ability to develop more 
outcome-based measures. Implementation of a ministry-wide strategy to 
strengthen the performance measurement framework will begin in August 
2003. With the implementation of the strategy, the Ministry will focus its 
performance measurements on outcomes and indicators that reflect the 
effectiveness of the programs. This plan is consistent with the government's 
overall performance measurement strategy. : 


As noted by the Provincial Auditor, the Ministry has implemented a number of 
initiatives that will help to improve agency accountability and assess 
effectiveness—for example, standardized common intake and assessment 
tools that document improvements in children. 





Program Co-ordination 
The Ministry’s April 1997 document entitled “Making Services Work For People” set out a 


new framework to reshape the social services system that would: focus on the needs of the 
individual and family; respond more quickly to those needs; and use resources more 
effectively. Proposed changes to the social services system included introducing the 
following: 


a single point of access for residential-based services; 


fewer access points for, and greater co-ordination of, other services to help families and 
individuals gain access to the services that are the most appropriate for their needs; 


a mechanism to provide integrated information about all available services; and 


a case resolution function, whereby those who need help the most receive essential 
supports first. 
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Since the time of our last audit, some of the changes have been implemented, partly in 
order to address previous audit recommendations made by our Office. These include 
centralized points of access for residential-based services, the co-ordination of services for 
highly complex cases, and case resolution functions within each region. These changes 
should help to improve the efficiency of the needs assessment and intake functions for 


agency services. 


CONTROL OF TRANSFER PAYMENTS TO 
AGENCIES 


Agency Funding Requests and Approvals 


In our prior audits of various ministry transfer-payment programs, including our most 
recent audit of the CFI program in 1997, we found that the Ministry lacked the necessary 
information to make informed funding decisions. Although the Ministry has consistently 
agreed that informed funding decisions require appropriately detailed budget submissions, 
we continue to have the same concerns we had in our 1997 audit of the CFI program. Our 
concerns with the agency budget submissions and the subsequent service agreements the 
Ministry entered into with agencies are as follows: 


¢ Program descriptions did not provide sufficient detail for the Ministry to assess the 
specific nature and levels of service to be provided. For example, in many cases, the 
Ministry could not assess the duration or intensity of services provided, both of which 
have a significant impact on program costs. 


e Many budget submissions combined a number of different programs into one budget 
request, which makes it impossible to determine and evaluate the reasonableness of the 


requested funding. 


¢ Some budget submissions contained questionable information that the Ministry did not 
assess for accuracy or reasonableness. For example, one budget request indicated that a 
secretary would be paid the equivalent of $178,000 per year. Even though we were 
advised by the Ministry that a secretary was not actually paid this amount, there was no 
evidence that the Ministry made any inquiries with respect to this information in the 
budget request. 


In most cases the Ministry continues to provide agencies with the same amount of base 
funding as was provided in the prior year without assessing the reasonableness of that 
amount. In many cases, it also provides funding for new initiatives without any needs 
assessment. As a result, similar services may be funded at significantly different levels, so 
funding inequities between agencies may be perpetuated; questionable items may be 
funded; and, after funding is provided for specific agreed-to services, it may be spent for 
other purposes. For example, information available at the Ministry indicated that: 
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The cost of residential Child Treatment care in the regional offices that we visited 
ranged from $271 to $589 per day, and the Ministry could not explain this variance. 


The Ministry provided two agencies with additional operational funding of $200,000 
and $167,274, respectively, to be used for building renovations. The Ministry not only 
did not assess the reasonableness of these amounts, but in providing the funding it 
bypassed its own capital expenditure planning and priority-setting process. 


At one regional office, four agencies were to provide early intervention services for 
children with autism and received funding totalling $1.9 million for this purpose. They 
spent all of this funding without providing the contracted-for services. The Ministry did 
not have the necessary detailed information to explain what services had been provided 


with this funding. 


The risk of not providing funding based on assessed need is that those agencies facing 


significantly increased service demands and cost increases may have to eliminate services to 


meet their budget. During our audit we noted that a number of agencies have reduced 
services in order to operate within their historical base funding allocation, which has 


contributed to the waiting-list problem. For example: 


In February 2001, a consultant conducted a review of the funding arrangements and 
operations of three agencies. The consultant concluded that these agencies were under- 
resourced and would have to cut service in order to address funding shortfalls. As a 
result of these financial pressures and in order to reduce costs, one agency had closed an 
eight-bed children’s residence. At the time of our audit, this agency had 10 children on 
its residential-based care waiting list. Agency staff estimated that these children would 


wait an average of 18 months before receiving any residential-based services. 


Another agency closed an eight-bed children’s mental health residence as of March 31, 
2001 in order to reduce its costs and operate within its funding allocation. 


Ministry staff we talked to acknowledged the importance of ensuring that transfer-payment- 
agency funding is reasonable and commensurate with the services received. However, they 


also pointed to several practical difficulties in linking funding to need for services, as follows: 


Demands on an agency’s services may fluctuate significantly from year to year. As a 
result, the services provided in a given year, along with the resultant expenditure 
surpluses or deficits, may not be relevant to the service and funding needs in the 
following year. 


A significant proportion of an agency’s costs are fixed and cannot be reduced even with 
year-to-year changes in service levels. 


Given that overall program funding is of a fixed amount, providing additional funding 
to one agency based on assessed needs would by necessity require providing another 


agency with less funding, which could be problematic. 
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While we acknowledge that these issues must be taken into consideration, we also stress that 
funding decisions must take into account changes in the demand for services in order to 
correct current inequities and prevent future ones. 








Recommendation 





In order to help ensure that agency funding is equitable and based on meeting 
the needs of children in every community, the Ministry should: 


e ensure that all agencies include sufficiently detailed, reliable, and relevant 
information in their program budget submissions; 


e assess all requests for funding and ensure that the amounts approved are 
commensurate with the demand for and value of the services to be 
provided; and 


e ensure that funding provided is spent for the purposes intended. 


Ministry Response 


The Ministry acknowledges the need for detailed information in program 
budget submissions. Through the annual transfer-payment budget package, 
the Ministry will reconfirm its requirements for agency funding requests and 
approvals, that is, that information included in budget submissions is 
sufficiently detailed, reliable, and relevant; that requests for funding amounts 
approved are commensurate with the demand for and value of services 
provided; and that funding provided is spent for the purpose intended. 
Implementation of a policy framework for children’s mental health services will 
also assist regional offices in identifying service expectations associated with 
funding. In addition, the Ministry is building on the financial skills of its 
program supervisors by providing training in financial management and 
accrual accounting. 





Annual Program Expenditure Reconciliations 


The Ministry requires that agencies submit an Annual Program Expenditure Reconciliation 
(APER), together with an audited agency financial statement, no later than four months 
after the end of the fiscal year. Ministry procedures require that the APER be reconciled to 
the agency’s total expenditures and its approved budget in order for the Ministry to identify 
any surplus funding. Effective April 1, 2000, Management Board Secretariat (MBS) 
required that ministries recover all surplus funds, and ministry policy requires that the 
recovery occur within 24 months of the year-end to which the surpluses relate. The 
Ministry is to use the audited financial statement to verify that the APER was completely 
and accurately prepared, and the APER must be approved within twelve months of the 
agency s year-end. 
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We continue to question, as we have done for many years, the effectiveness of the APER 
process. While we found that, for the 2000/01 fiscal year (the most recent year for which 
the process had been completed), the APERs had generally been received and approved on 
a timely basis, both the APERs and the audited financial statements lacked the necessary 
detail to identify expenditures that were inappropriate or ineligible for funding. As a result, 
the Ministry cannot verify that the APERs have been accurately completed and that surplus 
funding has been identified and returned. More specifically, we noted the following 


Concerns: 


¢ A majority of the audited financial statements submitted by agencies along with their 
APERs consolidated a number of different social service programs and represented the 
financial results of the agency as a whole. Ministry policy allows the submission of such 
statements only if they include a note detailing the funding surplus or deficit for each 
program receiving ministry funding. However, we found in our review of a sample of 
these financial statements that three-quarters of them did not contain the required note. 
Consequently, the Ministry could not use those audited statements to verify that the 
APERs reflected actual program expenditures. 


¢ All of the audited financial statements we reviewed were prepared based on a different 
basis of accounting (the accrual basis) than were the APERs (prepared on a cash basis). 
Therefore, the financial statements cannot be used to compare with the APERs and to 
verify that the APERs were correctly prepared. 


Accordingly, we continue to question the effectiveness of the review and approval process. 





Recommendation 


In order to strengthen its financial accountability process, the Ministry should 
ensure that Annual Program Expenditure Reconciliations (APERs) and audited 
financial statements contain sufficiently detailed and comparable information 
to allow for the detection of ineligible or inappropriate expenditure items and 
funding surpluses. Secondly, the Ministry should develop a more effective 
process for the review and approval of APERs. 


Ministry Response 


The Ministry acknowledges the need for strong financial accountability 
processes and has taken steps to improve the Annual Program Expenditure 
Reconciliation (APER) form and process. Improvements were made to the 
2001/02 APER form by referencing comparable information contained in 
audited financial statements and service contracts and providing a list of the 
most common ineligible expenditures. To improve the detection of funding 
surpluses, the rules for applying financial flexibility have been clarified and 
added to the 2001/02 APER form. 





To improve the APER review and approval process, in 2003/04 the Ministry is 
providing regional and corporate office staff with training on how to use the 
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APER more effectively. The government’s change to accrual accounting will 
also positively impact on the APER process. 


Surplus Recovery 


As noted above, ministry policy requires that all ministry-funded surpluses identified 
through the APER process now be returned to the Ministry within 24 months of the 
agency year-end to which they relate. All recovered surpluses are to be transferred to the 
government's Consolidated Revenue Fund. 


In our review of a sample of APERs for the 2000/01 fiscal year, we found that, for surpluses 
that agencies self-declared, the surplus recovery process was generally initiated within the 
required time frame. However, we also noted cases at two of the three regional offices we 
visited in which the process was circumvented and therefore not effective. 


Specifically, these offices entered into supplemental service agreements with their transfer- 
payment agencies that spanned two fiscal years. The terms of these agreements ranged from 
10 to 13 months and generally started one or two months prior to the end of a fiscal year. In 
our review of such agreements, we noted that the Ministry provided the full funding for 
these agreements at the time of their start date and recorded the entire expenditure in the 
fiscal year of the start date. However, agencies generally incurred most of the expenditures, 
and provided the majority of the agreed-to services, in the following fiscal year. Only at the 
end of the agreement—that is, only in the second fiscal year—were agencies required to 
account for this funding through the APER process. In effect, these agreements are a means 
of transferring ministry funding from one fiscal year to the next. They thus circumvent 
Management Board’s requirement and the Ministry’s process for identifying and recovering 


annual funding surpluses through the APER process. 


Recommendation 


The Ministry should not enter into service agreements that span two fiscal 
years, since doing so circumvents Management Board Secretariat’s 
requirement to identify and recover annual funding surpluses. 


Ministry Response 


The Ministry acknowledges this policy requirement and will reconfirm that 
agreements spanning two fiscal years require the identification and recovery 
of annual funding surpluses. The Ministry is training staff in the methods of 
accrual accounting, introduced government-wide in 2003/04, which also will 
help ensure the identification and recovery of surpluses. The Ministry has also 
added the rules for applying financial flexibility to the Annual Program 
Expenditure Reconciliation (APER) form and has clarified these requirements 
in the APER instructions. 
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Information Systems 


FINANCIAL AND SERVICE INFORMATION SYSTEM 


In our 1993 and 1997 audits, we reported that the Ministry did not have a management 
information system for monitoring financial and service data. Partly in response to our 
audits, the Ministry implemented a new management information system in August 1998 
that captures annual and quarterly financial and service information. 


However, we found in our current audit that the Ministry was not using the system to 
analyze whether programs are funded consistently across the province and whether the 
Ministry is receiving value for money for the services provided by the agencies. 


Furthermore, we found that, even if the Ministry were to use the system for such analysis, 
the system was of limited usefulness for the following reasons: 


¢ The definition of “individuals served,” which is the primary measure the Ministry uses 
to monitor the majority of agencies’ services, was interpreted differently at different 
agencies. For example, some agencies combine the number of individuals who began 
receiving service in an earlier period and are continuing to be served with the number 
of new cases; other agencies report only the number of new cases—individuals who 
have begun receiving service in the current period. As a result, these statistics cannot be 
used to analyze financial or service data between agencies. 


¢ Insome cases, the management information system either included inaccurate 
information or lacked the required information. For example, we noted that in a 
number of cases, an agency’s budget and service targets, rather than its actual fourth- 
quarter results, had been entered into the system. As a result, the system could not show 
variances between the budgeted and actual results, even when such variances existed. 


¢ A number of different services are not distinguishable from one another within the 
information system. As a result, the Ministry cannot determine the costs or the service 
outputs for any particular service. For example, within the CFI Non-Residential 
program (which accounts for over 40% of total Children’s Mental Health Services 
expenditures, or $127.7 million), services ranging from day treatment to drop-in-centre 
funding are not distinguishable in the system. In addition, the Ministry cannot compare 
agencies with respect to their provision of the same service since, when agencies provide 
more than one type of service, the cost and output data for all services provided are 
combined for those agencies within the information system. 


¢ Information in the system pertaining to funding for supplemental services provided 
under contracts spanning two fiscal years is not linked to the services’ actual costs and 
outputs. That is, as previously noted, the system shows the Ministry paying fully for 
supplemental services in one fiscal year, but the services are mostly provided in the 
following fiscal year. Also, in many cases, the funding given out for such contracts is not 
distinguished from the funding given out under the regular single-fiscal-year contracts. 
Asa result, the Ministry cannot determine from the system whether an agency 
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providing any of the supplemental services overspent or underspent its regular funding 
allotment. A second result is that the information system may show a significant, but 
misleading, variance between such an agency’s budgeted and actual results—the 
variance may be partially or entirely due to the mixing together of supplemental service 


funding information with regular fiscal-year data. 


AUTISM PROGRAM INFORMATION SYSTEM 


In the 2000/01 fiscal year, the Ministry developed an information system that contains 
financial and client information for the nine regional autism programs. According to the 
Ministry, this information is mainly used to help develop new policies for the autism 
programs. The system contains useful information such as the number of children receiving 
intensive behavioural intervention service, the number of children waiting for assessment 
and services and the corresponding waiting periods, and the ages of the children receiving 
services. However, we also noted the following: 


e ‘The system provides only consolidated province-wide information. As a result, the 
Ministry cannot relate specific data, such as lengthy waiting lists and long waiting times, 
to specific agencies. 


¢ Ministry staff stated that they were uncertain about the accuracy and completeness of 
the information in the system. 


Without sufficiently detailed and reliable data, the autism program information system is of 
limited usefulness to the Ministry. 





Recommendation 


The Ministry should ensure that its management information systems provide 
sufficiently detailed, relevant, and accurate information in order to help 
determine whether services provided by transfer-payment agencies are 
effective and represent value for money spent. 





Ministry Response 


The Ministry is committed to having information systems that provide the 
appropriate information to help determine the effectiveness and value for 
money of services provided by transfer-payment agencies. Since 1997, the 
Ministry has made significant efforts to enhance Service Management 
Information System (SMIS) data quality. The Ministry is currently implementing 
an action plan to improve the management of information on services provided 
by transfer-payment agencies that includes strengthening reporting timelines, 
providing infrastructure support, and revising data definitions. Because the 
Ministry is also using SMIS data more regularly in the development of policy/ 
program initiatives (for example, residential services for children with special 
needs), the Ministry is making a concerted effort to clean up these databases 
so that it can be certain that information is sufficiently detailed, relevant, and 
accurate. 
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MINISTRY OF COMMUNITY, FAMILY 
AND CHILDREN’S SERVICES 


3.03—Family Responsibility 
Office 





BACKGROUND 


Under the authority of the Family Responsibility and Support Arrears Enforcement Act, 1996, 
the Family Responsibility Office (Office) administers and enforces all court-ordered child 
and spousal support in Ontario, as well as court-ordered support in many other jurisdictions 
where the payers are resident in Ontario. It also enforces private separation agreements that 
are voluntarily registered with a court and filed with the Office. At the time of our audit, the 
Office administered approximately 180,600 family-support cases. We understand that this 
number represented about half of all marriage breakups in Ontario. Other separation 
agreements are normally handled without the assistance of the Office. 


Support orders and separation agreements may be voluntarily withdrawn from 
administration by the Office, as long as both parties involved agree to the withdrawal and 
complete the required forms. We noted that of the four other provinces whose family- 
support enforcement programs we contacted, three had an opt-in rather than a mandatory 
system, where all recipients had to voluntarily choose to have their support obligations 
enforced by the program. 


During the 2002/03 fiscal year, the Office collected approximately $561 million from 
support payers and forwarded a similar amount to support recipients (when we last audited 
the Office, in 1999, this amount was $500 million). As at March 31, 2003, support 
payments in arrears totalled approximately $1.3 billion, which represented an 8% increase 
since our 1999 audit. We also noted that approximately 23,000 support recipients, whose 
cases were in arrears totalling over $200 million, were receiving provincial social assistance. 


Responsibility for the Office was transferred from the Ministry of the Attorney General to 
the Ministry of Community, Family and Children’s Services effective April 9, 2001. The 
Office has approximately 400 staff, all of whom are located in one central office in Toronto. 
In addition, the Office maintains a panel of approximately 100 private-sector lawyers to 
provide family-support litigation services across the province. The Office's operating 
expenditures for the 2002/03 fiscal year were $28.3 million, of which $19.8 million (70%) 
was for salaries, wages, and benefits. 
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AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of the Family Responsibility Office were to assess whether 


adequate policies and procedures were in place to ensure that: 


* support orders were enforced effectively and receipts were accurately accounted for and 
distributed to support recipients on a timely basis; and 


° — services were delivered with due regard to economy and efficiency and the effectiveness 
of the services provided was monitored and reported on. 


Our audit included a review of the Office’s administrative policies and procedures, 
interviews with appropriate staff, and an assessment of a sample of case files and pertinent 
summary information and statistics. Information was also obtained from external parties, 
such as the Office of the Ombudsman, and from family-support enforcement programs in 
other jurisdictions. 


At the beginning of our audit, we identified the criteria that would be used to address our 
audit objectives. These criteria were reviewed and agreed to by senior Office management. 
Our audit work was primarily conducted during the period October 2002 to March 
2003, with emphasis on program policies and procedures in place during the 2002/03 
fiscal year. 


Our audit was conducted in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants. Accordingly, it included such tests and other procedures that we 
considered necessary in the circumstances. 


We did not rely on the Ministry’s Comprehensive Audit and Investigations Branch (internal 
audit) to reduce the extent of our audit work because it had not conducted any audit work 
at the Office during the past two years. 


OVERALL AUDIT CONCLUSIONS 


As was the case at the time of our last audit, in 1999, we concluded that the Family 
Responsibility Office did not have satisfactory systems and procedures in place for initiating 
contact and taking appropriate and timely enforcement action where payers were in arrears 
on their family-support obligations. In fact, it is our view that, unless the Office takes 
aggressive enforcement action, supported by effective case management and significantly 
improved information technology and communications systems, it is in grave danger of 
failing to meet its mandated responsibilities. We found that the Office’s services were 
impaired, and we had the following concerns: 


* Unlike most other provinces, which use a process of individual case management, 
Ontario does not assign each case to an individual caseworker. Therefore, no one 
individual has responsibility for or is held accountable for the administration of most 
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cases. In addition, although we were advised that in practice the Office has assigned 
cases with outstanding arrears greater than $50,000 to caseworkers since 2001, 
approximately 1,500 such cases, with arrears totalling $126.7 million, were not assigned 
at the time of our audit in November 2002 and were therefore not actively monitored 
or enforced. 


¢ Since 1994, the number of caseworkers has declined by 20%, whereas the number of 
cases has increased from 126,000 to 180,600. As a result, the average number of cases 
per caseworker has steadily increased. For example, the number of cases with 
outstanding work items assigned to senior caseworkers now ranges from 600 to more 
than 1,300, averaging 890 per caseworker. By comparison, the average caseloads in 
Quebec and Alberta were 400 and 335, respectively. The Office has not established 
criteria or standards for determining a manageable workload. This may well have been a 
key factor as to why, since 1994, arrears have increased by $600 million and cases with 
amounts in arrears have increased by 40,000. 


¢ The Office’s practice of commencing enforcement action only after being notified by 
recipients of non-payment resulted in unreasonable delays in enforcement. On average, 
seven months elapsed between the time support fell into arrears and the time the Office 
initiated the first enforcement action. 


¢ More than half the cases in arrears we reviewed had inordinately long gaps, often as 
long as two years, between enforcement actions. 


Staff efforts to enforce support obligations and to provide responsive services continue to be 
significantly hampered by the Office’s inability to develop and implement the necessary 
improvements to the computer system. We recommended in our 1994 report that the 
Office improve its computer systems and ensure that the deficiencies identified were 
corrected. In response to our recommendation then, and again in our follow-up report in 
1996, the Ministry stated that “the current computer system must be replaced.” However, 
the same computer system continues to be used even though it cannot provide timely and 
appropriate information to facilitate client service or management of the program. If the 
Office successfully implements the improved integrated service-delivery model and the 
integrated information management system outlined in the business case recently submitted 
to the Minister, many of our concerns could be addressed. 


We also found a lack of due regard for economy and efficiency inasmuch as: 


¢ anumber of cases with significant arrears were not assigned to caseworkers and 
therefore not actively enforced; 


* arrears were not aged to determine how long amounts have been outstanding and to 
assess their collectibility; and 


¢ almost 90% of telephone calls to the call centre made from outside Toronto were 
blocked and therefore not answered, and this required clients to call repeatedly in order 
to get through to have their questions answered. 
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Senior management of the Office was aware of a number of the needed improvements 
noted in this report and had prepared a business case outlining the current problems and 
proposed corrective actions. We understand that, at the end of our audit, the business case 
was being reviewed by the Minister of Community, Family and Children’s Services. 


In our review of the quarterly reports prepared for Management Board Secretariat and 
internal management of the Office, we found that they did not disclose information 
necessary to assess the efficiency and effectiveness of the Office. 


Our review of a sample of new family-support orders and separation agreements received by 
the Office noted that these cases were registered and collection action initiated within the 
targeted 30-day time period. In the financial transactions we sampled, we also found that 
the accounting controls over support payments received and disbursed were satisfactory. In 
most cases, support payments received were generally disbursed within 48 hours of receipt. 






















Overall Office Response 


The Office supports the Provincial Auditor's overall conclusions about the 
Family Responsibility Office and, as noted by the Provincial Auditor, has’ 
prepared a business case outlining many of the current problems and 
proposed corrective actions. However, the Office will be limited in its ability to 
address many of the specific audit recommendations with its current service- 
delivery model and limited supporting technology until a new service-delivery 
model with state-of-the-art supporting technology is implemented. 





With an increase of 1,200 to 1,400 new cases every month, the Office cannot 
meet many of the audit recommendations without significant change to its 
business model and systems technology. From the huge number of calls that 
do not get through to the call centre to the lack of technology to support bring- 
forward actions, the Office is trying to manage the high volume of inquiries 
and resolve client complaints as best it can. 





Should the Office receive approval from Cabinet to implement an improved 
service-delivery model and supporting technology, the Office should be able to 
realize significant improvements in both enforcement and customer service 
that will address many of the concerns outlined in the Provincial Auditor's 
Report. 


DETAILED AUDIT OBSERVATIONS 


Each year, the Family Responsibility Office registers approximately 18,000 new cases and 
closes about the same number of previously active cases. The Office’s goal is to register new 
cases within 30 days of receipt of all the required documentation, at which time filing 
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information packages are sent to the support payer and recipient, and in most cases support 
deduction notices are sent to the payer’s known income sources. 


Although payers may make support payments directly to the Office, in most cases support 
payments are withheld by income sources and are remitted by them to the Office. Support 
payments received by the Office are to be forwarded to the recipient within 48 hours of 
their receipt by the Office. When a payer is in partial compliance or non-compliance with 
his or her support obligations, the Office may take a number of progressively aggressive 
enforcement actions on receiving initial notification of non-payment or partial payment by 
the support recipient. 


At the time of our audit, approximately one-third of all payers were in full compliance with 
their support obligations, one-third were in partial compliance—defined as meeting at least 
85% of their current monthly obligations—and one-third were in non-compliance. We 
note that the definition of partial compliance has changed since the time of our last audit; 
therefore, a comparison of compliance rates is not meaningful. 


Detailed information about the status of their accounts with the Office is available to both 
support payers and recipients through the Office’s toll-free call centre. Limited account 
information can be obtained from an automated telephone inquiry line, and general 
information about the Office can be obtained from the Office’s Web site. 


On April 1, 2000, the Office introduced a number of fees as shown in the following table. 
Administration Fees 


Typeotree Amount 


Additional director's statement of arrears $25 per statement (after 
the first statement request) 


Postdated cheque $10 per cheque 


Adjustment to records after direct payment $100 

of support to recipient from payer 

Confirmation letters re: real estate $150 

transactions 

Aggressive enforcement process emote wane 00 
gg P in a 9-month period 


Source of data: The Family Responsibility Office 





We noted that for the fiscal year ended March 31, 2003, the total administrative fees 
collected by the Office were $744,366. 


ENFORCING SUPPORT OBLIGATIONS AND 
RELATED MATTERS 


Many of the following concerns were well known to senior management of the Office and a 
business case had been prepared outlining the current problems and proposed corrective 
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actions. At the end of our audit, we understand that the Minister of Community, Family 
and Children’s Services was reviewing the business case. 


Case Registration 


Requests to register and enforce family-support orders and separation agreements are 
received in the Office’s intake unit, where they are reviewed. If the information provided is 
complete and accurate, the Office’s goal is to register the case and initiate enforcement 
action within 30 days of receipt of the information. Our review of a sample of case files 
registered during the 2002/03 fiscal year found that in approximately 90% of the cases the 
initial information received was complete and accurate. Generally, these cases were 
registered and enforcement was initiated within 30 days. 


Support orders and separation agreements that are incomplete or that contain contradictory 
information cannot be registered and are instead returned to the sender for completion or 
clarification. However, the Office did not have adequate procedures in place to ensure that 
the required information or clarification was received back on a timely basis and to follow 
up when it was not. In these instances, registration and enforcement were delayed on 
average for about three-and-one-half months, and in some cases for as long as eight to ten 
months. Such situations can create serious financial hardship for the affected recipients who 
rely on the Office to ensure they receive their support payments. 


Recommendation 


To ensure the Office fulfills its responsibilities to collect and forward support 
payments to families, it should ensure that it receives all the required 
information for registering and enforcing support obligations on a timely basis 
and promptly initiate follow-up action when it does not. 


Office Response 


In many cases, the Family Responsibility Office does not have full control over 
all the information it needs to register and enforce support obligations in a 
timely manner because it necessarily relies on others to provide it with the 
required information. However, the Office will continue to develop outreach 
plans to work with the appropriate stakeholders (e.g., courts, lawyers, and so 
on) who have responsibility for forwarding case registration information to the 
Office. 


Currently, consistent and timely follow-up by the Office is still strongly 
hampered by the lack of technology that supports bring-forward notes and 
automatic reminders. Should the Office receive approval to implement the 
proposed new service-delivery model and supporting technology, it is 
expected that there will be significant improvement in case registration 
processes and in the ability to ensure prompt follow-up on any outstanding 
information that is required before the Office can take action. 
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The Office will also undertake a review and redesign of its filing package to 


help clients better understand what information is required and to make it 
easier for them to fully complete the package. 





Document Scanning 


The Office’s intake unit receives approximately 1,300 faxes and 2,300 pieces of mail 
relating to active cases per day. The Office requires that all these documents be scanned. 
Scanned documents are to be available electronically to all caseworkers for case 
administration purposes and for answering inquiries through the Office’s call centre. 


We found, however, that the document-scanning and retrieval process was not working as 
intended for the following reasons: 


¢ In some cases, documents that should have been scanned were not. 
¢ Some scanned documents were of such poor quality that they were unreadable. 


¢ The Office’s scanning equipment cannot scan blue ink, but the documents sent out by 
the Office for respondents to complete do not request that blue ink not be used. 


¢ The computer system that provides caseworkers access to scanned documents is typically 
out of service for about one hour per day and this results in lack of access to required 
information when caseworkers answer client inquiries. 


These document problems are particularly detrimental to the productivity and efficiency of 
staff assigned to the call centre. 


Recommendation 


In order for all necessary case documentation to be available on a timely basis 
for administering cases and for answering telephone inquiries, the Office 
should ensure that: 

e all necessary case documentation is scanned; 

e scanned documents are of an acceptable quality; and 

e system downtime is minimized. 


Office Response 





The Family Responsibility Office is aware that the current document-scanning 
and retrieval process is a problem. Last year, in order to address the issue, the 
Office undertook a rigorous review process for acquiring a new document- 
scanning system. Following Management Board of Cabinet guidelines, the 
Office made a recommendation for replacing the current system with a system 
that is much faster and more efficient and meets the business needs of the 
Office. 
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A vendor has been chosen and the new document-scanning system should be 
in place in fall 2003. Successful implementation of the new system will include 
appropriate training, as well as ensuring that appropriate policies and 
procedures are in place and adhered to by staff. 





Case Management Model 


As it did at the time of our last audit, in 1999, the Office continues to manage its caseload 
on an “issue management” basis. In this method, any caseworker can answer inquiries with 
respect to a particular file and perform simple tasks—for example, processing an 
information update such as an address change. However, certain tasks that require in-depth 
knowledge of the case and potential follow-up at some future date can be performed only 
by senior caseworkers who temporarily assume exclusive jurisdiction over that case until the 
issue is resolved. 


Although the issue management model has some obvious advantages, such as providing 
maximum flexibility in responding to case inquiries and processing the many routine 
information updates that are often necessary, our view continues to be that it has a number 
of significant shortcomings that include the following: 


¢ Since the majority of cases are not assigned to caseworkers, no one individual has 
responsibility for administering them. In that regard, we note that, although we were 
advised that in practice the Office has assigned cases with outstanding arrears greater 
than $50,000 to caseworkers since 2001, approximately 1,500 such cases, with arrears 
totalling $126.7 million, were not assigned at the time of our audit in November 2002 
and were therefore not actively monitored or enforced. 


¢ Since a caseworker in effect voluntarily assumes responsibility for a case as a result of 
fielding an inquiry or complaint, assigned caseloads vary significantly. They range from 
about 600 cases to, in one instance, more than 1,700 cases per caseworker. 


° Many caseworkers often worked on the same case over a period of time. For example, 
our review of a sample of case files found that on average seven different caseworkers 
worked on the same case over a period of five years. In one instance, 34 different 
caseworkers worked on the same file over the same five-year period. 


We understand that Ontario is one of two provinces that have not adopted a case 
management service-delivery model where a specific caseworker is assigned to each case. The 
benefits of assigning a caseworker to each file are that the performance of all caseworkers in 
collecting arrears can be monitored and ongoing familiarity with the case details facilitates 
effective collection action. 
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Recommendation 


To help ensure that effective and timely enforcement actions are taken, the 
Office should review its case management practices and consider assigning 
the responsibility for each case to an individual caseworker. 


Office Response 


Since 1996, the Family Responsibility Office has operated under an “issue 
management” system, rather than a case management system. The Office 
recently completed a comprehensive feasibility study with an intensive review 
of alternative case management options. In particular, the case management 
systems of other enforcement jurisdictions were reviewed to determine if they 
would work here in Ontario. 


As a result, the Office has developed a proposal to implement an Integrated 
Service Delivery Model—a comprehensive case management mode/—that 
includes integrated teams for providing client service. The Ministry is currently 
reviewing the proposal for this model, and it is hoped that the proposal will be 
approved by Cabinet shortly. 





Caseloads 


A manageable caseload is a prerequisite for caseworkers to administer family-support cases 
adequately. At the time of our audit, the Office had approximately 160 staff who worked 
part-time on enforcement and case administration and part-time in the Office’s call centre. 
Specifically, 100 of these were senior staff members who were more experienced and 
answered calls for three hours per day and 60 were junior staff members who answered calls 
for four-and-one-half hours per day. The time not spent answering calls was used primarily 
for individual case administration and enforcement. 


Even though high caseloads were a concern at the time of our 1999 audit, the average 
number of cases per caseworker has steadily increased over the years as detailed in the 
following table. 


Number of Cases per Caseworker, 1994-2003 









eo March 2003 March 1999 March 1994 


180,600 | 170,000 | _ 126,000 


Source of data: The Family Responsibility Office 
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In addition, the number of case files assigned to senior caseworkers with outstanding work 
items (see the next section on “Bring-forward Notes”), generally ranged from approximately 
600 to more than 1,300, and averaged 890. 


Although increased caseloads may be justifiable when there have been improvements in 
business processes or information technology support, this has not been the case in Ontario 
over that time. We also note that in comparison, similar family-support enforcement 
programs in Quebec and Alberta had an average caseload of 400 and 335 cases per 


caseworker, respectively. 


Recommendation 


To help improve the administration of family-support cases in a timely and 
effective manner, the Office should establish criteria and standards for 
manageable caseloads and staff accordingly to ensure that the standards are 
met. 


Office Response 


As previously noted, the Family Responsibility Office has reviewed alternative 
service-delivery models and recognizes that there must be a move towards a 
case management system with supporting technology (rather than an issue — 
management system). This move is pending the Ministry’s decision on the 
options provided by the Office. 





Under the proposed case management system, enforcement officers would 
have cases assigned to them and be directly responsible for those cases over 
the long term, rather than working in a call centre capacity as they do now. This 
would ensure that each case is followed through from beginning to end and 
that the appropriate follow-up is completed. The shift to a case management 
model would assist in ensuring manageable caseloads as enforcement officers 
would be able to spend their time focusing on enforcement and not on call — 
centre shifts. A small call centre to handle general inquiries would still operate. 


Should the Office receive approval from Cabinet to implement the proposed 
case management model and supporting technology, the Office will develop 
appropriate criteria and standards for manageable caseloads. With 1,200 to 
1,400 new cases each month, the Office believes that the proposed system 
would help to realign resources and create more manageable caseloads. 





Bring-forward Notes 


As a result of either a client inquiry or enforcement action undertaken, caseworkers often 
enter a bring-forward note into the computerized case-file information system. The bring- 
forward note is an automatic reminder to a caseworker that the inquiry or enforcement 
action needs to be followed up on, usually within 30 days of placing the note on file. 
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Our review of outstanding bring-forward notes for a sample of caseworkers found that for 
many there was no follow-up on a timely basis. The number of bring-forward notes that 
were not followed up on within the required 30 days ranged from 46 to more than 800 per 
caseworker, and averaged more than 300. In addition, many of the outstanding bring- 
forward notes were more than a year old, indicating that required action had not been 


taken for a very long time. 





Recommendation 













To help ensure that client inquiries and enforcement actions are dealt with 
appropriately, the Office should ensure that all caseworkers conduct the 
necessary follow-up work on a timely basis. 





Office Response 


The Family Responsibility Office is committed to ensuring timely follow-up of 
client inquiries and enforcement actions. Only when a new case management 
model with supporting technology is implemented will the appropriate tools 
(for example, bring-forward notes and automatic prompts) be in place to 
ensure timely follow-up for each case. The proposed technology will also 
provide better monitoring reports for management to use to ensure that 
caseworkers conduct the necessary follow-up work. 












Support Enforcement Action 


At the end of 2002, approximately 136,000 or three-quarters of all active cases registered 
with the Office were in arrears. Arrears totalled approximately $1.3 billion, an 8% increase 
from the time of our 1999 audit. Since 1994, arrears have increased by $600 million and 
cases with amounts in arrears have increased by 40,000. A breakdown of the amount in 
arrears by amounts outstanding for individual accounts is provided in the following table. 
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Total Cases with Amounts in Arrears as at December 31, 2002 





Source of data: The Family Responsibility Office 


We note that although the Office did not track how long the amounts in arrears had been 
outstanding, either individually or in total, many were thought to be more than a year old. 


‘Taking the appropriate enforcement action on a timely basis is essential if the Office is to 
effectively collect support payments in arrears. In conducting enforcement action, 
caseworkers are to follow a series of steps prescribed in the Office’s “enforcement tree.” The 
tree begins with a series of passive steps that escalate progressively to more aggressive steps as 
outlined below. 


Initial passive enforcement steps include: 


¢ permitting the payer to enter into a voluntary arrears payment schedule to pay off 
arrears owing; | 


° obtaining federal garnishment of 100% of the payer’s income tax refund; 


¢ obtaining a writ of seizure and sale to secure any proceeds in the event of a disposal of 
assets by the payer; 


* intercepting lottery winnings; and 

* reporting the payer to the credit bureaus. 

More aggressive enforcement steps include: 

¢ garnisheeing the payer’s bank account; 

* suspending the payer’s driver’s licence; 

* suspending federal licences and passports; 

¢ taking the payer to court for a default hearing; and 


* transferring the debt to a private collection agency. 
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Based on the timelines established for each step in the process, we estimated that from 
beginning to end, enforcement action ought to be completed in approximately one-and- 
one-quarter years. 


However, our review of a sample of case files for which enforcement actions were taken 
found that the steps were often neither timely nor effective. For example: 


¢ The Office’s practice of commencing enforcement action only after being notified by 
recipients of non-payment resulted in unreasonable delays in enforcement. On average, 
seven months elapsed between the time support fell into arrears and the time the Office 
initiated the first enforcement action. In one case, enforcement action was initiated 
18 months after the case first went into arrears. 


¢ More than half the cases we reviewed had inordinately long time gaps between 
enforcement steps. Time gaps of two years were common. 


¢ On average, it took the Office three-and-one-half years to complete the entire 
enforcement cycle, and in one case it took eleven-and-one-half years. 


These delays in enforcement actions often made it difficult to locate the payer and collect 
the amounts owed to the support recipients. In addition, there was no evidence that 
supervisory staff reviewed the case files or were even aware of the time delays in conducting 


the required enforcement measure. 






Recommendation 


To help ensure the effectiveness of its enforcement actions in collecting 
support arrears, the Office should: 















e identify accounts in arrears on a more timely basis and initiate contact with 
the defaulting payer as soon as possible; 

e adhere to the established timetable for the prescribed enforcement steps in 
a timely manner; and 

¢ ensure supervisory staff monitor case files for compliance with the 
prescribed steps and established timetable and where necessary take 
corrective action. 





Office Response 


The Family Responsibility Office has in place policies and procedures to 
ensure the effectiveness of its enforcement actions. However, consistent and 
timely follow-up of enforcement and compliance is hampered by the current 
issue management business model and lack of appropriate supporting 
technology. 






The Office has implemented a strategy to increase aggressive enforcement 
action. Currently, all arrears over $50,000 are assigned to individual client- 
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services associates until the cases are in compliance. Supervisors regularly 
monitor and act on these cases to ensure timely enforcement. 


The Enhanced Collection Agency Project is another initiative, in which the age 
of arrears is used to determine which cases will be referred to the project. 


Enhanced Collection Agency Project 


In October 2001, the Office entered into a three-year agreement with four private 


collection agencies to pursue collection efforts in up to 40,000 cases where support 
payments were not received in over six months and all of the Office’s prescribed efforts to 
enforce payment were ultimately not successful. 


Under the terms of the agreement, the agencies are paid a commission based on funds 
collected in excess of the payer’s ongoing monthly support obligations. In all cases, the 
applicable commission is added to the payer’s outstanding arrears balance. 


At the end of January 2003, about 18,750 cases with arrears totalling $290 million had 
been transferred to the collection agencies. Approximately $2 million net of commissions of 


about $400,000 has been collected by these agencies and transferred to the Office. 


In our review of a sample of files transferred to the collection agencies, we found that they 
all met the criteria for transfer. We also noted that agency commissions were correctly 
determined. 


Payment Processing 


For the 2002/03 fiscal year, the Office processed on average 7,600 transactions totalling 
approximately $2.4 million daily. About 60% of all payments received by the Office were 
made through manually prepared remittances, usually cheques, and about 40% were 
remitted electronically. More than 80% of all disbursements were issued through direct 
deposit to the recipient’s bank account. The Office’s targeted turnaround time from the 
time the payment is received to the time of disbursement is 48 hours. 


In our review of a sample of financial transactions, we found that the accounting controls 
over support payments received and disbursed were satisfactory. In most cases, support 
payments received were disbursed within 48 hours of receipt. 


IDENTIFIED SUSPENSE ACCOUNT 


When the recipient is known, there are two primary reasons why some receipts may not be 
disbursed within 48 hours. Financial reasons, such as the need to pro-rate a receipt between 
two related accounts or the need to wait for a payer's cheque to clear the bank, and 
enforcement reasons, such as the need to wait for a varied court order, may delay payment 
to recipients. Such receipts are temporarily held in the identified suspense account. 
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At the end of January 2003, the balance in this account was $1.7 million. Office policy 
requires that finance staff clear and disburse most receipts held for financial reasons within 
24 to 48 hours (10 days for payments awaiting a cheque’s bank clearance). For amounts 
held for enforcement reasons, caseworkers are to resolve and transfer amounts held within 
90 days. Receipts that cannot be transferred within 90 days need to be reviewed again 
within every 90 days thereafter. 


In our review of a sample of items in this account, we found that: 


¢ almost half the receipts held for enforcement reasons were not cleared within 90 days 
and were held on average for about three years—in one case, a receipt had been held 


for almost 10 years; and 


¢ half the receipts held for financial reasons were not released within the required 48- 
hour time frame; in fact, on average, these receipts were held for five months. 


UNIDENTIFIED SUSPENSE ACCOUNT 


Receipts that cannot be readily attributed to a specific case are temporarily held in the 
unidentified suspense account for further investigation. Funds are held in the suspense 
account until such time that they can be disbursed to the appropriate recipient. 


At the end of January 2003, the balance in this account was $1.75 million. Office policy 
requires that finance staff initiate an investigation within 48 hours of receiving funds for 
which the intended recipient is not clear, so that this recipient can be identified and the 
money forwarded. This may involve searching the office database or contacting the person 
or institution remitting the funds. There is no specific requirement to follow up on items 
that cannot be resolved by this first investigation, and these receipts are held in the account 
until the necessary information is received. 


We reviewed a sample of items in this account and found that in most cases unidentified 
receipts were investigated, identified, and forwarded to the intended recipient within 48 
hours. However, documentation supporting the identification of the intended recipient was 
often lacking and there was no evidence of management approval of the release of funds. 
Given the need for very strong internal controls over this area, we believe these weaknesses 
represent a high-risk area for the Office. It is therefore essential that subsequent 
disbursements need to be well controlled. 


We also noted that an aging of the remaining items in the account found that 
approximately $1 million (58%) of the balance was over one year old. 


Recommendation 


To ensure that internal controls are strengthened and that all support 


payments received are forwarded to the intended recipient on a timely basis, 
the Office should: 
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follow up on and resolve all items in both the identified and unidentified 
suspense accounts on a timely basis; 
¢ adequately document the basis on which initially unidentified receipts were 
identified and management approval of the release of such funds. 





Office Response 


The Family Responsibility Office processes over 95% of payments received 
within 48 hours. Only 0.3% of the total payments received in a year represents 
items in both the identified and unidentified suspense accounts. Monies are 
usually diverted into suspense accounts for reasons beyond the Office’s 
control (for example, the need to wait for a decision of the courts, 
correspondence from lawyers or response from an income source that 
indicates some sort of change, no case identification, and so on). The funds are 
released as quickly as possible from the suspense accounts after required 
information is returned to the Office. The Office continues to ensure that 





payments in suspense accounts are processed as quickly as Is practical. 


As the Provincial Auditor indicates, the majority of the unidentified funds 
received by the Office are adequately researched and validated, according to 
the Office’s existing policies and procedures, before the funds are released. 
The Office will commit to regular management control checks to ensure that 
| the policies are being adhered to. 








Interest on Arrears 
Both our 1994 and 1999 audit reports noted that the Office did not calculate and pursue 


interest on support arrears provided for under the terms of many support orders, even 


though the Office is legally responsible for enforcing all aspects of a support order. 


In response to our 1999 recommendation that the Office calculate and pursue interest on 
all arrears where interest is required under the applicable court order, the Office indicated 
that its computer system was unable to calculate and accrue interest owing and that it was 
not efficient or economical for caseworkers to calculate manually the amount of interest 
owing, It is still the Office’s view that this continues to be the case, although we understand 
that recipients are now encouraged to file annual statements for interest owing on arrears. 


As a result, interest on arrears continues to be pursued only if the recipient calculates the 
amount of interest owing and provides the amount owing to the Office in a sworn 
statement. 
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Recommendation 


To help ensure compliance with support orders and to encourage prompt 
payment from payers, the Office should compute and charge interest on 
arrears for those cases where the court orders stipulate that interest is 
applicable. 


Office Response 


This is not anew recommendation for the Family Responsibility Office, and it 
continues to be the Office’s view that, until the Office has new supporting 
technology, it is not practical to calculate interest on arrears. 


Thus, the responsibility for calculating the interest rests with the recipients, 
who have the information concerning when payments were due, the dates 
payments were made, and any resulting interest accrued. 


Customer Service 
Prior to December 1996, the Office operated out of eight regional offices and each office 


provided customer services, including counter service, to area clients. Effective December 
1996, the Office closed its regional offices and consolidated all its operations into one 
centralized office in Toronto. 


Since that time, the Office has operated a toll-free call centre that is the primary means for 
clients to communicate with the Office, in addition to written correspondence. The call 
centre’s hours of operation are Monday to Thursday from 8 a.m. to 7 p.m. and Friday from 
8 a.m. to 5 p.m. There are separate telephone numbers for calls originating within and 
outside the Toronto calling area. We understand that Ontario’s call centre is open for more 
hours than call centres in other provinces. 


Although the Office monitors some information with respect to the call centre—the 
number of calls answered, average waiting times for answered calls, and the average length 
of each call—it does not regularly monitor the number of calls blocked (that is, calls 
receiving a busy signal). A one-time study of blocked calls for the three months March to 
May 2001 inclusive found that the number of calls blocked were substantial as shown in 
the following table. 
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Number of Telephone Calls, March—May, 2001 
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Approximately 130 caseworkers answer calls for an average of about four hours per day. 
Each caseworker is expected to answer six calls per hour. As a result, the call centre ought to 
be able to answer about 3,000 calls per day, substantially fewer than the 16,500 calls that 
are attempted. The Office’s telephone system allows up to 66 calls to be placed in a queue to 
await the next available caseworker. Any additional calls receive a busy signal and are in 


effect blocked. 


Even though repeat calls accounted for some of the calls blocked, there is no question that 
the call centre is not adequately resourced to answer all calls within a reasonable time. As 
well, fewer calls were answered than expected because on average: 


e about 14% of the caseworkers were not available for answering calls because of medical- 
related reasons, such as stress or illness; 


* caseworkers who staffed the call centre did so for on average a half-hour less per day 
than they should have; and 


¢ caseworkers in fact answered 3.6 calls per hour, significantly fewer than the expected six 
calls per hour, so that rather than the targeted 3,000 calls per day being answered, only 
1,900 calls on average are answered. 


In contrast, we placed calls to the call centres for several other provincial family-support 
programs and found that on average our calls were answered within five minutes. Similarly, 
we noted that the Australian family-support enforcement program implemented a service 
standard requiring that 90% of incoming calls be answered within two minutes. According 
to a recent audit report of the Australian program, this standard is consistently met. 


Recommendation 


Since the call centre is the primary means whereby clients communicate with 
the Office, the Office should review its call centre operations and take the 


steps necessary to ensure that all calls are answered or responded to within a 
reasonable period of time. 
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Office Response 


The Family Responsibility Office acknowledges that there is a large number of 
calls that do not get through. However, it should be noted that the Office has 
taken a number of steps to foster improvements in this area. These include: 


hiring a call centre co-ordinator; 

introducing a call monitoring process, along with monitoring and coaching 
workshops; 

preparing a “monthly snapshot” report combined with more frequent 
performance meetings; and 

introducing a complaint and compliment tracking form. 


However, the Office is limited in its ability to improve the number of clients 
gaining access to the call centre, particularly because of the increasing 
caseload each year. Should the Office receive approval to implement a new 
service-delivery model and supporting technology, clients will have direct 
access to assigned caseworkers, poe call centre backlogs and increasing 
client satisfaction. 





Call Centre Alternatives 


Clients of the Office can also access very limited information about their accounts through 
the Office’s automated telephone system, or they can obtain general program information 
and download various program forms from the Office’s Web site. The automated telephone 
line and the Web site operate seven days per week, 24 hours per day. We noted that from 
April to December 2002 inclusive, on average about 400,000 calls were made each month 
to the automated telephone line. 


However, the automated telephone line and the Web site are limited in their usefulness 
because they are not interactive. For example, clients cannot request account statements or 
report information changes related to their cases. They also cannot communicate with 
caseworkers about any aspect of their accounts, which would reduce the number of calls to 


the call centre. 


Our research of a number of jurisdictions in Canada found that two provinces (Alberta and 
British Columbia) had interactive Web sites that enabled clients to report changes in their 
personal information or access a wide range of information about their account. 


Recommendation 


To help alleviate the demand for information and services through the Office’s 
call centre, the Office should consider expanding access to detailed account 


information and the range of services available through the automated 
telephone line and Web site. 
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Office Response 


The Family Responsibility Office is implementing a Personal Identification 
Number (PIN) project to provide more information and protected access for its 
clients who call the automated-voice information line. 


The PIN project was a recommendation of the Office of the Information and 
Privacy Commissioner (IPC), and has been undertaken with the support of that 
office. As the PIN project is implemented, the Family Responsibility Office will 
continue to work with the IPC. 


The recommended approach is a phased implementation of the PIN initiative. 
Phase 1, the first of three phases, will provide access to the automated-voice 
information line, where callers who have been assigned PINs (specifically, 
support recipients and support payers) can obtain more personal information 
than what is currently provided. Phase 1 will also provide the foundation for 
future phases (2 and 3), in which agents will be able to verify callers’ identity 
using their PIN and which will see the expanded information from the 
automated-voice line also made available on the Internet using PIN access. Full 
implementation of the PIN initiative will be a component of the new Integrated 
Service Delivery Model. 





Computer System 


The Office's current computerized information system was initially developed in the mid- 
1980s primarily for registering cases and as a bookkeeping system to track receipts from 
payers and disbursements to recipients. The system, known as Maintenance Enforcement 
Computerized Accounting (MECA), was developed in COBOL using an IBM database at 
a time when the caseload was about 53,000 (it is currently about 180,000). In 1989, the 
system was enhanced to include certain case management information and reporting 


modules that are still used. 


By the 1990s, limitations in the computer system posed considerable problems in the 
Office’s consolidation and transition to a call centre-based business model. As a result, the 
Office added a front-end interface to the computer system—Family Responsibility Office 
New ‘Technology (FRONT)—whose primary purpose was to retrieve the most frequently 
sought information from the existing computer system and post it onto more user-friendly 
screens so that client inquiries through the call centre could be dealt with more efficiently. 
However, because FRONT is still dependent on MECA, which is now almost 20 years old, 
many of the primary computer system’s performance problems remained. Many of these 
have been known for a long time and include: 


¢ Programming enhancements take considerable time and are expensive, partly because of 
poor or missing system documentation. In fact, in many cases the Office is reluctant to 
program system changes for fear of making the whole system unstable. 
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e Since only limited information is available in FRONT, caseworkers often still have to 
undergo a cumbersome and time-consuming process to obtain desired case information 


from MECA. 


¢ Management is not provided with the necessary information for monitoring and 
assessing whether the program is delivered efficiently or effectively. For example, 
required information about the many aspects of case administration by caseworker or 
aging of amounts of support in arrears simply isn’t readily available. 


Nine years ago, we recommended in our 1994 report that the Office improve its computer 
systems and ensure that the deficiencies identified were corrected. In response to our 
recommendation then, and again in our follow-up report in 1996, the Ministry of the 
Attorney General stated that “the current computer system must be replaced.” 


In response to our recommendations in 1999, the Ministry of the Attorney General stated 


that the Office would address the performance of the computer system by: 
¢ doing an architectural review of the MECA system once the Y2K freeze was over; 


¢ hiring a consultant to do an availability and reliability audit of the whole system, and 
make recommendations; and 


* upgrading the document-imaging software and database server in order to handle the 
higher volume of users. 


At the time of our audit, there had been no substantial changes made to the computer 
system. As noted earlier, the Office is well aware of the computer system’s limitations and has 
submitted a business case for acquiring a new computerized integrated information system 
in conjunction with a new service-delivery model. 


Recommendation 


We urge that the process of implementing the needed computer support for 
the Office’s operations be significantly accelerated. 


Office Response 


The Family Responsibility Office has made improvements in the performance 
of its current computer system but needs to replace the system in order to 

keep up with the demands of Office operations. A new service-delivery model 
to meet the Office’s business needs, supported by new technology, is needed. 


The Office undertook a business planning review and feasibility study of 
systems requirements and prepared recommendations for a new case 
management model with supporting technology. This submission is currently 
under review for approval and funding. 
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PERFORMANCE MEASUREMENT 


Reporting Program Effectiveness 


The mandate of the Office is to ensure compliance with all registered family-support 
obligations and where necessary take appropriate enforcement action while treating both 
payers and recipients fairly. By assessing and reporting on the achievement of this mandate, 
the Office can demonstrate its effectiveness and can identify areas that need improvement. 


The Office prepares monthly and quarterly internal management reports, as well as a 
quarterly report that is submitted to Management Board Secretariat. Although the Office 
does not prepare a report specifically for the Ministry of Community, Family and Children’s 
Services, we understand that the Ministry is provided with a draft copy of the report 
prepared for Management Board. 


The reports generally provide statistical indicators for the following: 
e the number of cases under administration; 


¢ the total amount of support payments collected and in arrears, as well as the compliance 


rate; 
¢ the number of telephone calls answered; 
¢ the type and number of enforcement actions taken; and 
¢ the number of cases assigned to the Ministry. 


The preceding information gives the reader a general sense of the type and volume of 
activity undertaken by the Office; however, it does not provide the data necessary to assess 
the effectiveness of the Office and to identify areas in need of improvement. Examples of 
required information for these purposes include: 


e the number of cases with significant arrears not assigned to a caseworker and therefore 
not subject to enforcement action; 


e the timeliness of enforcement actions taken on assigned accounts; 


¢ the number of telephone calls to the call centre that were blocked and therefore not 
answered; 
* the aging of support arrears and an assessment of their collectibility; and 


¢ the nature and number of complaints received from all sources. 


We also note that the only publicly disclosed performance measure is the number of 
accounts in full or partial compliance with their support obligations. For both the 2000/01 
and the 2001/02 fiscal years, the actual compliance rate of 67% was.slightly below the 
targeted rate of 71%. 
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Recommendation 


To help ensure and be able to assess whether family-support obligations are 
effectively enforced and that areas in need of improvement are identified, the 
Office should measure and report on additional results indicators, such as: 


e the number of cases with significant arrears not assigned to a caseworker 
and therefore not actively enforced; 


e the timeliness of enforcement actions taken on assigned accounts; 


e the number of telephone calls to the call centre that were blocked and 
therefore not answered; 


e the aging of support arrears and an assessment of their collectibility; and 
e the nature and number of complaints received. 


Office Response 


The Family Responsibility Office agrees with this recommendation but is 
limited in its ability to address these issues with the current system and 
information technology. Should the Office receive approval to implement a new 
service-delivery model and supporting technology, the new system will 
generate a number of relevant management reports that will facilitate timely 
responses. 


Over the past few years, the Office has worked with all the other jurisdictions 
throughout Canada that enforce family-support obligations to develop an 
“Operational Policies and Goals” document. All have signed and committed to 
this document, which will provide for standardized communications processes 
and co-operative principles across jurisdictions as related to enforcement 
guidelines and actions. Currently, the jurisdictions are setting out baseline data 
to identify performance measures so that guidelines and standards can be 
developed. 





Assessment of Client Satisfaction 


Complaints and client surveys are two important tools to determine client satisfaction and to 
assess the effectiveness of the program. We reviewed the processes surrounding both. Based 
on our review, we concluded that the Office was not assessing and monitoring client 
satisfaction to identify required service improvements. 


COMPLAINTS 
Complaints received directly by the Office were entered into individual case-file notes; 
however, the total number and the nature of the complaints received were not separately 


logged and tracked. 
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For the fiscal year 2001/02, there were 17,000 complaints received via other sources, such 
as the offices of the Minister and Deputy Minister of the Ministry of Community, Family 
and Children’s Services (1,874), the local MPP offices (14,000), and the Office of the 
Ombudsman (1,126). 


Different response time guidelines exist for complaints depending on their source. Written 
complaints received directly by the Executive Director in the Family Responsibility Office 
must be addressed and responded to within 15 days. Written complaints received through 
the Minister’s office must be responded to within 30 days. Referrals from the local MPP 
offices and the Office of the Ombudsman must be responded to within five days. 


Based on our review of a sample of complaints and the related process, we found that 
complaints were generally responded to within the required time frames. However, we also 
found that the Office did not summarize and analyze the complaints received by the nature 
of the complaint with a view to identifying areas in need of improvement. In that regard, we 
contacted the Office of the Ombudsman and were advised that many of the complaints 
were concerning delays in case administration and inappropriate enforcement of support 
orders. 





Recommendation 


To help increase client satisfaction and the effectiveness of services provided, 
the Office should: 


e log complaints from all sources to ensure that all complaints are 
addressed; and 


e categorize and analyze the complaints received from all sources to identify 
areas most in need of improvement. 


Office Response 


The Family Responsibility Office is reviewing its process for addressing 
complaints. Although all complaints are logged, the Office is presently limited 
in its ability to categorize and analyze all types of complaints, as current 
technology does not facilitate comparison and analysis. Should the Office 
receive approval to implement a new service-delivery model and supporting 
technology, the new system technology will generate the reports needed to 
monitor complaints and identify areas for improvement. | 











CLIENT SATISFACTION SURVEYS 


According to the government's common service standards, customer feedback should be 
collected regularly to improve services and increase clients’ satisfaction. One method of 
obtaining feedback is through a client satisfaction survey. 
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The last client satisfaction survey conducted by the Office was in the spring of 1999. Our 
review of this survey showed that many of the concerns identified then, such as long call- 
centre wait times and the need for proactive case monitoring and improved enforcement, 
have not been resolved, as noted in our current observations. No other client survey or 
customer feedback has been collected since 1999. 

















Recommendation 


To aid in the assessment of both customer satisfaction and effectiveness of 
services provided, the Office should regularly conduct client satisfaction 
surveys that identify areas that are working well and those in need of 
improvement. 


Office Response 


The Family Responsibility Office is committed to conducting a client service 
survey in fall 2003 and using the results as a benchmark for future-year 
reporting on improvements to client satisfaction with service. The first survey 
will be conducted prior to the implementation of the new case management 
model in order to set baseline data. The Office will continue to collect customer 
feedback on a regular basis via client satisfaction surveys as a part of the 
business plan to improve services and increase client satisfaction. 
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MINISTRY OF CONSUMER AND BUSINESS 
SERVICES 


3.04—Policy and Consumer 
Protection Services Division 





BACKGROUND 


The mandate of the Policy and Consumer Protection Services Division (Division) of the 
Ministry of Consumer and Business Services is to oversee business and other practices in the 
Ontario marketplace. It does so by establishing fair marketplace and technical standards, 
educating the public about standards and other relevant issues, monitoring businesses and 
enforcing their compliance with the various laws and regulations that exist for the protection 
of consumers, managing relationships with service-delivery partners, and developing 
strategic policy and legislation. The legislation for which the Division monitors compliance 
includes, for instance, the Business Practices Act and the Consumer Protection Act, 2002. 


In the 2002/03 fiscal year, the Division had approximately 100 staff and operating 
expenditures of approximately $9.7 million. The Division consists of three branches: the 


Policy Branch, the Marketplace Standards and Services Branch, and the Sector Liaison 
Branch. 


The Policy Branch is responsible for policy development across the Ministry. The 
Marketplace Standards and Services Branch, which accounts for approximately two-thirds 
of the Division’s expenditures, administers various statutes relating to consumer protection 
and business licensing. The Marketplace Standards and Services Branch carries out its 
mandate of ensuring public safety and consumer protection through its registration and 
licensing activities in a number of industries. Some of the types of businesses that need to be 
registered or licensed include collection agencies, theatres, and cemeteries. The Branch also 
processes consumer complaints, inspects businesses for compliance with consumer 
protection acts, investigates alleged infractions, and undertakes initiatives to educate the 
marketplace. 


Since 1997, the Ministry has also delegated the administration of a number of consumer 
and public-safety statutes to eight delegated administrative authorities (delegated 
authorities). The delegated authorities are not-for-profit corporations, led by representatives 
within each industry, that carry out the day-to-day functions of ensuring public safety and 


consumer protection by regulating and monitoring business practices in their industry. 
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Nevertheless, the Ministry retains overall responsibility for the outcomes of the delegated 


authorities’ activities in protecting the consumers and the public. 


The delegation of authority from the Ministry to the delegated authorities was facilitated by 
the Safety and Consumer Statutes Administration Act, 1996. The Division’s Sector Liaison 
Branch is responsible for overseeing the following eight delegated authorities: 


¢ the Technical Standards and Safety Authority; 
¢ the Electrical Safety Authority; 

¢ the Ontario Motor Vehicle Industry Council; 
¢ the Ontario New Home Warranty Program; 

¢ the Real Estate Council of Ontario; 

¢ the Travel Industry Council of Ontario; 

¢ the Board of Funeral Services; and 


¢ the Vintner’s Quality Alliance of Ontario. 


AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit were to assess whether the Division had adequate procedures 
and systems in place to: 


¢ ensure compliance with the legislation and ministry policies that are established to 
ensure public safety and consumer protection; and 


¢ measure and report on the effectiveness of the key services and programs that it delivers 


in fulfilling its mandate. 


At the beginning of the audit, we identified audit criteria that would be used to address our 
audit objectives. These were reviewed and accepted by senior ministry management. Our 
audit was performed in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants, and accordingly included such tests and other procedures as we 
considered necessary in the circumstances. 


The scope of our audit, which was substantially completed in April 2003, covered the 
Marketplace Standards and Services Branch and the Division’s monitoring of four delegated 
authorities. Our work included reviews and analyses of the Ministry’s administrative 
procedures and guidelines, interviews with staff, and reviews of case files. As current 
legislation does not provide our Office with the authority to audit the delegated authorities, 
our work was confined to reviewing administrative agreements and performance 
information obtained by the Ministry, as well as, in some cases, additional information 


volunteered by the delegated authorities. We did not rely on the Ministry's Internal Audit 
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Services Branch to reduce the extent of our work because they had not recently conducted 
work within the scope of our audit. 


OVERALL AUDIT CONCLUSIONS 


We concluded the Ministry needed to enhance its procedures and systems to ensure 
compliance with legislation and ministry policies, to better protect consumers, and to better 


ensure public safety. 
With respect to the Marketplace Standards and Services Branch, we had the following 


concerns: 


¢ The Ministry did not deploy its inspection resources based either on an assessment of 
risk to the general public or to consumers of specific services, or on the number of 
complaints it recetved. For example, the practices of debt collectors had been the 
number one source of complaints received by the Ministry in the past five years. While 
it received approximately 4,000 complaints and inquiries (with almost 800 formal 
written complaints) about debt collectors in 2001/02, the Ministry conducted fewer 
than 10 inspections. In contrast, for theatres and video retailers, about which the 
Ministry received only eight complaints, the Ministry devoted over 95% of its inspection 
resources to inspecting this industry: almost 1,600 inspections were conducted to check 
whether video retail stores were operating with a valid licence and were selling adult 
videos only with proper stickers indicating their ratings. 


¢ Of the 5,000 cemeteries in Ontario, fewer than 2,000 had filed their required annual 
returns with the Ministry. (When individuals purchase cemetery plots, cemetery owners 
are required to deposit at least 40% of the purchase price in a trust account to ensure 
resources are available to properly care for and maintain cemetery grounds—the status 
of each trust account is to be included in the required annual return.) Our audit 
showed that even for cemeteries that had filed their 2001 returns, the Ministry had not 
processed or performed an adequate review of them to ensure proper accounting for 


the trust funds. 


¢ The Ministry did not ensure that the outcomes of its regulatory activities were captured 
in its management information system. For example, for the approximately 6,000 
inspections performed over the past three years, the results of only 800 were recorded in 
the system. Such information would help management monitor the effectiveness of 
ongoing activities and identify recurring problems. 


With respect to the Ministry’s oversight of the delegated authorities, we concluded that the 
Ministry did not have adequate assurance that public safety and consumers were properly 
protected by the industry oversight organizations. Also, the Ministry’s governance and 
accountability arrangements with the delegated authorities were not of sufficient strength to 
ensure appropriate monitoring of the delegated authorities’ performance. Specifically, we 
noted the following concerns: 
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¢ The Ministry had not ensured that data on the outcome of delegated authorities’ 
activities, as reported by delegated authorities—such as the number of safety-related 
incidents, the number of serious injuries, and the number of complaints recetved—were 
reliable. For example, the Technical Standards and Safety Authority (TSSA) indicated 
that some safety-related incidents were under-reported because businesses might worry 


that reporting incidents would result in their operations being inspected by the TSSA. 


¢ The Ministry’s monitoring of inspections, investigations, and other enforcement 
activities undertaken by delegated authorities in response to violations they identified 
was inadequate. For example, of the 4,600 natural-gas pipeline incidents reported in 
2001, about 170, or 4%, were investigated by the Technical Standards and Safety 
Authority; the Ministry had not assessed whether this level of investigation was sufficient 
and appropriate to protect public safety. 


¢ The Ministry was unable to obtain adequate information to monitor the activities and 
the outcomes of the activities of the Ontario New Home Warranty Program (ONHWP) 
because the Ministry had not, as yet, been able to negotiate a formal agreement with the 
ONHWP that would outline the Ministry’s requirements. The information the Ministry 
had about this delegated authority’s activities was mostly compiled from the ONHWP’s 
published annual report. The Ministry therefore had little information with which to 
assess whether new homeowners were being properly protected, especially given that the 
ONHWVP had stopped reporting complaints information three years before. 


¢ The boards of directors of the various delegated authorities did not reflect a balanced 
representation of interests. To maintain balanced representation between industry and 
public interests, the Minister can appoint 50% of the members on delegated authorities’ 
boards of directors. This provision would ensure that the governing boards of the 
delegated authorities are reasonably independent and not dominated by members 
appointed by the industry being regulated. Our review of the composition of the boards 
of the delegated authorities showed a significant under-representation of public interest. 
In addition, attendance at board meetings by Ministry staff appointed as government 
representatives was generally below that of other board members. 


DETAILED AUDIT OBSERVATIONS 


MARKETPLACE STANDARDS AND SERVICES 
BRANCH 


To allow the Ministry to monitor the marketplace and thereby protect the interests of 
consumers, businesses in certain industries must be registered with the Ministry. For 
instance, the following parties must obtain a registration or licence from the Ministry before 
they can engage in the regulated business: collection agencies, bailiffs, credit reporting 
agencies, theatres, and cemeteries. Registration provides the Ministry with the information it 
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needs to monitor their activities to ensure the honesty, integrity, and financial responsibility 
of an industry and to ensure that consumers are protected against unfair or substandard 
business practices, such as misrepresentation, overcharging for home renovations, pressure 
sales, and upfront fee scams. 


The Marketplace Standards and Services Branch’s monitoring of the Ontario marketplace 
includes: processing and issuing registrations and licences, regulating industry practices, 
establishing minimum standards, verifying the qualifications and requirements of 
participants who can operate in a specified sector, and monitoring that all parties are 
complying with legislation and policies that exist to protect consumers. 


On a yearly basis, the Branch carries out its mandate of ensuring public safety and 
consumer protection by processing about 20,000 registrations and over 30,000 consumer 
complaints and inquiries and by conducting over 1,600 inspections and almost 300 
investigations. As well, the Branch can initiate corrective action when it identifies cases of 
non-compliance. Such action can include: revoking licences or registrations, laying charges, 
and pursuing prosecutions and convictions. 


Following Up On Consumer Complaints 


The Consumer Protection Bureau Act gives the Ministry the mandate to “receive and 
investigate complaints of conduct in contravention of legislation for the protection of 
consumers.” Based on the complaints it receives from consumers, the Ministry carries out 
inspections to determine whether further ministry action, such as investigation and 
enforcement, is required. The Ministry can also use inspections to proactively educate 
businesses and thereby reduce non-compliance with statutes and deter unfair practices. 
Complaints relating to industry sectors that are regulated by one of the eight delegated 
authorities are referred to the relevant delegated authority to handle. 


Excluding those complaints that are referred to delegated authorities, the Ministry receives 
around 27,000 telephone complaints and inquiries per year. In most cases, consumers are 
advised to first try to resolve their complaint directly with the business, and they are 
informed of steps they can take themselves, such as going to small claims court. In cases 
where ministry follow-up action is requested by consumers, they have to file a complaint in 
writing to authorize the Ministry to take further action. The Ministry receives about 5,000 
written complaints per year. 


While the Ministry cannot force businesses to compensate consumers, based on complaints 
it has received, the Ministry can initiate inspections of the companies being reported. The 
numbers of ministry inspections and complaints received for the 2001/02 fiscal year, as well 
as the numbers of inspections conducted, are outlined in the following table. 
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Complaints and Inquiries Received by the Ministry’ 
and Ministry Inspections by Industry, 2001/02 


Telephone Total 

Complaints Written Complaints | Inspections % of Total 
Flare) Complaints Elale| Conducted Inspections 
Inquiries Inquiries 


Collection agencies 3,340 768 : 4,108 uel Ee 


Credit reporting 
agencies 


Industry 





1 Excludes complaints and inquiries referred to delegated administrative authorities. 


2 Includes businesses like door-to-door sales; home renovations and repairs; and fitness centres, modelling 
agencies, and organizations offering prepaid services. 


Source of data: Ministry of Consumer and Business Services 


Our review revealed that the Ministry did not deploy its inspection resources based either 
on any formal assessment of risk to the public and consumers, or on the number of 
complaints received for each of the industries it monitors. 


For example, as shown in the table, the Ministry devoted over 95% of its inspection activities 
to theatres and video retailers, even though there were virtually no complaints about this 
industry. Most of the inspections were conducted to ensure video retailers were operating 
with a valid licence and were selling adult videos only with stickers to indicate their ratings 
by the Ontario Film Review Board. With respect to these inspections, we also noted that the 
ministry inspectors had seldom seized videos or DVDs, as permitted by current legislation, 
to deter repeated non-compliance. Instead, where instances of non-compliance were noted, 
more inspections of the same retailers were conducted. The Ministry indicated that, since 
the 2002/03 fiscal year, it had initiated the issuing of orders to seize about 700 videos. 


In contrast, the practices of industries that were at the top of the Ministry’s list of complaints 
and inquiries were seldom inspected. Inspections of collection agencies was one example. In 
the 2001/02 fiscal year, the Ministry conducted nine inspections in the industry even 
though it received 3,340 telephone complaints and inquiries and 768 formal written 
complaints against the industry. Of those written complaints, almost one-third related to 
three collection agencies, but none of those collection agencies were inspected that year, nor 
had they been inspected in the previous seven years. 


In 2001, the Ministry agreed with recommendations made by an outside consultant to 
devote at least half of its inspection efforts to responding to complaints and use the 
remaining half for proactive inspections, allocating inspection resources equally between 
collection agencies, cemeteries, and theatres. Nevertheless, at the completion of our audit in 
2003, most of the inspection resources were still devoted to inspecting video retailers. 
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In addition, we noted that when unscrupulous practitioners had been identified, the 
Ministry was not taking timely enforcement action to protect consumers. For example, we 
noted concerns with two bailiffs cases. Through bailiffs, creditors or commercial landlords 
can take possession or repossess private properties from a debtor or commercial tenant 
without having to go to court. In 2003, for the first time in the past five years, the Ministry 
proposed to revoke two bailiffs’ appointments, but only after it had received a number of 
written complaints about these two bailiffs abusing their authority in taking possession of 
properties—complaints dating back to October 1999 for one bailiff and to March 2001 
for the other. Despite the history of complaints and the fact that both bailiffs had been 
charged by the police—one in March 2000 and the other in May 2002—the Ministry did 
not propose to revoke these bailiffs’ appointments until the beginning of 2003. Ministry 
legal counsel had indicated to the Ministry that it does not have to wait for criminal 
convictions before proposing to revoke the appointment of a bailiff if there is evidence that 
the licence should be revoked to protect the public. 


Subsequent to our audit, the Ministry informed us that one of the above proposals resulted 
in the bailiff being successfully convicted and his licence being revoked; the other proposal 
was being appealed. 


In November 2002, the Ministry hired an investigator on contract to review complaints 
related to collection agencies. Subsequent to our audit, we were informed that this review 
resulted in the Ministry issuing a number of cautions, charges, and proposals to suspend 
and revoke 16 licences. While the Ministry was increasing its efforts to step up monitoring 
in this sector, it is important that such efforts be systematic and sustained and that they be 
expanded to include all high-risk sectors with large numbers of complaints. 


While inspections are used to monitor compliance with fair business practices, educate 
businesses, and deter non-compliance, investigations are a formal means for the Ministry to 
determine whether further enforcement action, such as prosecution, needs to be initiated. 
Most investigations the Ministry conducts result from complaints it receives, but other 
factors that may prompt an investigation might include information obtained from the 
police or new cases discovered during the course of an original investigation. We concluded 
that the investigations the Ministry had conducted were generally performed in a 
satisfactory manner. As well, where mediations were possible, we noted evidence that 
appropriate efforts had been made to obtain restitution for consumers. 


Recommendation 


To adequately protect the public, the Ministry should allocate its inspection 


resources for monitoring various industries based on a systematic assessment 
of risk as well as on the number of complaints it receives about these 
industries. : 
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As well, the Ministry should ensure unscrupulous practitioners are removed 
from the marketplace on a timely basis to protect consumers and the public 
from potential losses and abuse. 


Ministry Response 


The Ministry has developed a risk-based framework for allocating Marketplace 
Standards inspection resources based on advice from an independent risk- 
management consultant. The framework has since been implemented and will 
be followed to ensure that inspection resources are appropriately deployed. 


Where there is clear evidence of an offence, the Ministry is committed to taking 
timely enforcement action. 


Monitoring Cemeteries’ Trust Accounts 


One of the industries the Ministry regulates is cemeteries, which are monitored by the 
Ministry for compliance with certain financial requirements. There are over 5,000 
registered cemeteries in Ontario. To ensure public safety and consumer protection, 
cemetery owners are required to deposit a percentage (between 40% and 100%) of the 
sales of plots into trust fund accounts. The income from these accounts is intended to 
support the cost of caring for and maintaining the cemeteries (headstones, grounds, and 


buildings). 


Given that hundreds of millions of dollars have been paid by consumers and deposited into 
trust funds, cemetery owners are required to file annual returns, which include trust 
account statements, within three months after the cemetery’s fiscal year-end. The returns 
must be certified by the owners and the trustees to be complete and accurate. Cemetery 
owners are also required to file, within six months after the cemetery’s fiscal year-end, 
audited financial statements for care and maintenance trust funds containing more than 
$500,000 and for pre-need trust funds (those relating to pre-arranged burial services) of 
more than $100,000. 


Ministry staff are responsible for reviewing, on a timely basis, the annual returns from 
cemeteries owners to detect errors, omissions, and instances of non-compliance with filing 
requirements. The timely monitoring of such returns, especially relating to trust fund 
accounts, is intended to minimize the risk of payments by consumers not being deposited 
into trust accounts or an insufficient amount being deposited. The other risk is of a cemetery 
not having the resources needed for care and maintenance purposes. In such cases, the 
cemetery may have to be turned over to be operated by the local municipality at the expense 
of local taxpayers. 


Our review of the Ministry’s cemetery registration records found that a significant number 
of the registration records were missing. Ministry staff indicated that many of these records 
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were inadvertently lost or misfiled in 1992 during the transfer of responsibilities from one 
section of the Ministry to the current branch. 


In addition, we found that a significant number of cemeteries had not filed their annual 
returns. [he Ministry indicated that 2,000 of the 5,000 registered cemeteries were either 
not conducting any more burials or receiving fewer than 10 burial permits per year. Due to 
limited resources, since 1995, the Ministry has exempted these owners from filing the 
required returns. However, the main reason for requiring annual returns is to ensure 
sufficient funds are available for the care and maintenance of cemeteries. By exempting 
these owners, the Ministry could not ensure sufficient funds were available, as many of these 
cemeteries were established prior to the requirement of having trust accounts. These 
cemeteries in fact pose significantly higher risks than cemeteries with proper trust funds 
established for care and maintenance. 


As well, in our review of the 2001 filings for active cemeteries, we found: 


¢ The Ministry indicated that it had not received the returns due from 400 cemetery 
owners. We noted that the Ministry had not taken timely action to follow up on the 
status of these returns. For instance, only seven cemeteries had been inspected from 
1998/99 to 2001/02, and mostly as a result of complaints. Our review indicated that 
three owners of these active cemeteries had not been submitting annual returns to the 
Ministry for seven to eight years. One of these owners was prosecuted only in December 
2001 for failing to maintain a cemetery, which was then turned over to be operated by 


the local municipality. 


¢ The Ministry could not determine what caused delays in processing the returns that had 
been filed by 450 owners, other than speculating that the delays might be related to 
problems with the information received and the lack of ministry staff resources. 


¢ The Ministry did not perform proper reviews of the returns that it had processed. For 
example, for two-thirds of the returns processed, the cemetery owners and/or trustees 
did not report a balance for their trust fund accounts. The Ministry did not follow up 
on the reasons for the missing balances. We requested that the Ministry ask two of the 
owners to resubmit their trustee statements. These resubmitted statements showed a 


total of $300 million being held in trust. 


The lack of timely monitoring of the filing of annual returns from cemetery owners 


increases the risk that payments by consumers might not be properly deposited and 
accounted for in trust fund accounts as required to maintain and care for cemetery grounds. 






Recommendation 





To comply with legislative requirements relating to cemeteries in Ontario and 
to make sure that sufficient funds are available for the proper care and 
maintenance of cemeteries, the Ministry should ensure that: 
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cemetery registration records are complete and annual returns are filed by 
all cemetery owners within the required time frame; and 


ministry staff verify, on a timely basis, the balances of trust accounts 
established to care for and maintain cemeteries. 


Ministry Response 


The Ministry is reviewing its procedures to ensure that cemetery records are 
requested from registrants on a timely basis and that where material 
underreporting occurs, timely follow-up action is taken. The Ministry is 
examining its procedures for reviewing trust account balances for the 
approximately 2,500 owners of the 5,000 cemeteries who may be required by 
statute to maintain these funds. 


Measuring and Reporting on Effectiveness 


In April 1999, the Ministry implemented a new management information system to track 
complaints, inspections, and investigations. The system has the capability to generate reports 
that capture the Branch’s activities and the outcomes of those activities. At our request, the 
Ministry was able to generate the information outlined in the following table. | 


Marketplace Standards and Services Branch—Activities and Outcomes, 2000-2002 


2000 «~=—sé«< tt” 







2,079 2419 | 1,666. || | -20 
272 250 










Charges laid 









336 533 
[ient|an sr Pos | 
$1,283,080 | $954,570 | $461,830 


Source of data: Ministry of Consumer and Business Services 


742 
401 


We noted that although the Ministry's management information system was capable of 
generating data on the above activities, the Ministry was not ensuring that the outcomes of 
such activities were captured in the system. For example, for the approximately 6,000 
inspections performed over the past three years (as shown in the above table), we noted that 
the outcomes of only 800 of these inspections were recorded in the system. Such 
information would help ministry management monitor the effectiveness of ongoing 
activities and identify recurring problems. It would also enable the Ministry to report on the 
effectiveness of its activities and outcomes. 
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With respect to measuring performance related to inquiries and complaints, in the 2001/02 
fiscal year, the Ministry conducted a customer satisfaction survey of the Ministry’s handling 
of inquiries and complaints. The survey was conducted over the telephone with callers who 
had contacted the Ministry. The results indicated that the majority of callers surveyed (90%) 
rated the services they received to be good or excellent. However, we had the following 
concerns: 


¢ The survey was not independent given that it was done, at the end of the call, by the 
same staff who handled the inquiry or complaint. The risk is that callers who were not 
satisfied with the service may not answer honestly and staff may not objectively report 
the callers’ responses. The Ministry indicated that in future it intended to engage an 
independent company to conduct the customer survey. 


* Callers were asked at the end of the call whether they were satisfied with the courtesy, 
helpfulness, and timeliness of the service, but in cases where further action by the 
Ministry would likely be required, callers were probably not in a position at that point to 
assess whether the staff were effective in helping them resolve their concerns during the 
course of the telephone call. 


In addition, we noted that there was no survey done on written complaints. Such a survey 
would be an important and perhaps more objective indicator of the Ministry’s effectiveness 
in dealing with consumer concerns. 


Recommendation | | 


To enhance management’s ability to properly measure and report on its 
effectiveness in protecting consumers and public safety, the Ministry should: 


° use its management information system to capture and analyze the 
outcomes of its activities and thereby be in a position to improve and 
report on its effectiveness; and 


e conduct proper consumer satisfaction surveys of both ep and 
written complaints. 





Ministry Response 


The Ministry recognizes the importance of recording data pertaining to the 
outcomes of various activities, including investigations and inspections. With 
the successful implementation of the final phase of the new CATS business 
system in April 2001, the Ministry's ability to capture key data improved 
significantly. The Ministry will work to further improve its outcomes data 
capturing and processing as part of the ongoing refinement of the CATS 
information system. 





As noted in the Auditor’s Report, staff in the Marketplace Standards and 
Services Branch conduct regular customer satisfaction surveys at the point of 
transaction with consumers. To provide a further source of customer 
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satisfaction data, the Ministry retained an independent consultant to survey a 


sample of consumers who had dealings with the Ministry. It is expected that 
these two data sources will provide the Ministry with useful feedback. 





DELEGATED ADMINISTRATIVE AUTHORITIES 


‘Traditionally, the government has been responsible for protecting consumers and public 
safety in the Ontario marketplace by establishing relevant legislation and carrying out 
regulatory activities. In recent years, however, Ontario has established a number of 
delegated administrative authorities (delegated authorities) to permit certain industries to 
undertake regulatory functions. 


The delegated authorities are not-for-profit corporations with Boards whose members 
represent both the industry in question and the public interest—some members are 
nominated by the related industry and others are appointed by the Minister to represent 
the government, the public, and consumers. The Safety and Consumer Statutes 
Administration Act, 1996 was passed to facilitate the administration of a number of 
consumer and public-safety statutes by these delegated authorities. 


Under this model, the delegated authorities are responsible for all day-to-day administrative 
activities, including registration and licensing, complaints processing, inspections, 
investigations, and other enforcement functions. The provincial government is responsible 
for legislating the regulated industries, and the Ministry remains accountable for the 
outcomes of improving public safety and consumer protection by: 


¢ — setting policy direction, rules, and standards for regulated industries through legislation, 
administrative agreements with delegated authorities, and other oversight mechanisms; 


* ensuring adequate accountability frameworks and governance structures are in place— 
including those that allow for monitoring and taking corrective action, when 
necessary—to protect the public interest and promote public safety and consumer 
protection; and 


¢ providing the public with relevant, accurate, and timely information on the outcomes of 
the delegated authorities’ regulatory activities. 


For our review of the delegated authorities model, we selected two of the delegated 
authorities that are responsible for public safety in Ontario, the Technical Standards and 
Safety Authority and the Electrical Safety Authority, and two that are responsible for 
consumer protection, the Ontario Motor Vehicle Industry Council and the Ontario New 
Home Warranty Program. For the selected delegated authorities, we reviewed their 
administrative agreements with the Ministry and performance information obtained by the 
Ministry, as well as, in some cases, additional information provided by the delegated 
authorities. 
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A key reason that the delegated authorities were established was that they can be more 
responsive to the needs of the marketplace because of their interest in the industries they 
regulate. We noted that there was a workable accountability framework in place to delegate 
regulatory functions to certain industries. However, based on our review of the information 
gathered by the Ministry and of the Ministry’s monitoring activities, we concluded that the 
Ministry could not have adequate assurance that public safety and consumers were being 
properly protected. Our audit indicated that the Ministry did not have proper mechanisms 
in place to ensure outcomes reported by the delegated authorities were reliable. We also 
found that the Ministry’s monitoring efforts were inadequate. 


Our observations for the selected delegated authorities are outlined in the following 
sections. 


Technical Standards and Safety Authority 
In 1997, the Technical Standards and Safety Authority (TSSA) was delegated the authority 


to regulate safety in such sectors as amusement devices; boilers and pressure vessels; elevating 
devices; natural gas, petroleum, propane fuels, and related equipment; and upholstered and 
stuffed articles. It administers and enforces public-safety laws in these sectors under 
Ontario's Technical Standards and Safety Act. 


The Ministry has established performance indicators to measure and report on the 
outcomes of the TSSA’s activities to enable it to assess the TSSA’s effectiveness in promoting 
public safety. These indicators include: the number of reported incidents; the number of 
serious injuries and fatalities; and the number of complaints recetved—all of which are 
reported on annually by the TSSA in a publicly available report entitled State of Public 
Safety Report. 


When reported outcomes are to be used to assess effectiveness and to provide direction in 
addressing public-safety risks, the reliability of the reported outcomes is essential. Our review 
of information relating to the period 1998 to 2001 (the latest statistics available at the time 
of our audit) indicated that the Ministry had not ensured that the safety outcomes reported 
by the TSSA were reliable. Our specific observations are discussed in the following sections. 


Reported Incidents 


The safety-related incidents reported by the TSSA since 1998 are outlined in the following 
table. 
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TSSA’s Safety-related Incidents by Sector, 1998-2001 


% Increase/ 
Sector 1998 1999 2000 2001 Decrease 
Since 1998 


eS ee 

ae 

ee 
483 


Boilers and pressure vessels 


51 
533 
44a 


Source of data: TSSA’s 2001 State of Public Safety Report 


In the case of safety-related incidents for amusement devices and ski lifts, the statistics were 
showing a decreasing trend but TSSA management reported that the decrease was due to 
an increasing trend of operators not reporting all incidents. The TSSA acknowledged that 
incidents were likely under-reported in other areas as well because businesses might worry 


that reporting would result in their operations being inspected by the TSSA. 


In addition, we found that: 


Except in the case of fuel incidents, over the years, neither the TSSA’s annual reports nor 
its State of Public Safety reports indicated how many of the reported incidents were 
investigated. The reporting of both reported and investigated incidents would 
demonstrate the extent to which the TSSA had taken action on safety concerns 
reported. Also, reported incidents did not separate minor incidents from those requiring 
a direct compliance response. 


With regard to fuel incidents, the 444 cases that were reported by the TSSA for 2001 
actually related to the number of cases that were investigated by the TSSA in that year. 
Our review indicated that, in fact, 6,000 fuel incidents were reported to the TSSA in 
2001. Of this total, 4,614 were natural-gas pipeline incidents that resulted mostly from 
damages to underground pipelines. 


The TSSA reported in 2001 that the 4,614 natural-gas pipeline incidents noted above 
represented a slight increase over the 4,568 pipeline hits in 2000. However, our review 
of the 2000 State of Public Safety Report showed that the number of natural-gas pipeline 
incidents reported in that year was actually 3,446. No explanation was provided for why 
the 2000 number was revised upwards by over 1,000 from 3,446 to 4,568 between 
2000 and 2001. The Ministry was not aware of the significant difference until we 
brought it to its attention. 
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Serious Injuries 

In May 2002, the TSSA revised the serious injuries data it had reported for elevators, 
escalators, and moving walks, as well as for ski lifts, since 1998, citing classification errors. 
The statistics for these sectors for 1998 through 2001 are outlined in the following table. 


TSSA’s Reported Serious Injuries by Selected Sector, 1998-2001 (Revised Data) 


Decrease — 
Since 1998 © 


Elevators, escalators, and moving walks 


Source of data: TSSA’s 2001 State of Public Safety Report 





We reviewed the original reports from previous years and noted that without the revisions to 
numbers dating back to 1998, the safety trend would have been different. As outlined in 
the following table, the originally reported serious injuries numbers for these sectors revealed 
that instead of a slight increase (3%) in serious injuries for elevators, escalators, and moving 
walks, the increase would have been 70%; and instead of a decline of 20% in serious 
injuries relating to ski lifts, the data showed an increase of 300%. 


TSSA’s Reported Serious Injuries by Selected Sector, 1998-2001 (Pre-revised Data) 


_% Increase | 
_| Since 1998 


Elevators, escalators, and moving walks 


Ski lifts 





Source of data: TSSA’s 2000 and 2001 State of Public Safety Reports 


Since the number of serious injuries was a key indicator used by the Ministry to monitor 
safety outcomes, the lack of accurate information hindered the Ministry’s ability to direct the 
TSSA to take more timely action to ensure public safety. 


Fatalities 

In prior years, the TSSA reported data on fatalities separately from data on serious injuries, 
but in 2001, the TSSA combined fatalities and serious injuries in its State of Public Safety 
Report. For our audit, we requested segregated data. 


Our review of those data indicated that in the TSSA’s 2001 report, the combined data of 
fatalities and serious injuries were under-reported in two sectors. Specifically, the following 
figures were omitted: four fatalities in the boilers and pressure vessels sectors in 2000 and 
2001 and five fatalities in the fuels sector in 2000. 
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Complaints 


The Ministry had established reporting requirements relating to the tracking of complaints 
for all the delegated authorities. In its annual reports since 1998, the TSSA has indicated 
that it is committed to timely and responsive handling of complaints from the public and its 
industry clients. Complaint statistics from the TSSA are outlined in the following table. 


TSSA’s Reported Complaints, 
1998/99-2001/02 


Number of 


i‘ Complaints 


1998/99 


Sources of data: TSSA’s Annual Reports, 
1998/99-2001/02 





Ministry management indicated to us that despite the numbers reported above, the 
complaints did not, in fact, dramatically increase from 11 in 2000/01 to 995 in 2001/02. 
Rather, it explained that the TSSA had not been accurately tracking and reporting the 
number of complaints. The Ministry indicated that it had expressed concerns with the low 
numbers of complaints reported by TSSA; this resulted in better reporting in 2001/02. 
While we were unable to assess whether subsequent complaints data were more accurately 
reported, we were concerned that it had taken four years for the Ministry to act on its 
concerns. 


MONITORING ACTIVITIES AND OUTCOMES 


The Ministry is responsible for monitoring the activities and outcomes of the TSSA. The 
Ministry requires the TSSA to report on its inspection and enforcement activities on a 
quarterly basis. 


The Ministry indicated to us that the TSSA had recently developed a risk-assessment model 
for the deployment of inspection resources. In addition, the TSSA had adopted a more 
co-operative approach in working with sectors—one that was focused on education, 
training programs, and information seminars as means of encouraging compliance—in 
addition to using an enforcement-oriented approach that used prosecutions or other 
penalties. | 


Nevertheless, inspection activities are effective only if appropriate follow-up action is taken 
when violations are identified. In this respect, we noted that the Ministry had not assessed 
whether the nature and extent of follow-up action taken by the TSSA was appropriate. 
Specifically, while the Ministry knew how many inspections had been conducted, the 
Ministry had not monitored the results of inspections to assess their effectiveness in 


protecting public safety. 
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To illustrate by way of an example why it is important for the Ministry to monitor the 
effectiveness of the results of inspections, we noted that there were more than 4,600 natural- 
gas pipeline incidents reported to the TSSA in 2001, some of which resulted in fatalities. 
Our review showed that about 170, or 4%, of the incidents were investigated by the TSSA. 
However, the Ministry had not monitored and analyzed whether the TSSA’s level of 
investigation and other enforcement measures were sufficient and appropriate, nor what 
corrective action, if any, would be needed to reduce the number of incidents and the 
occurrence of fatalities. 


At the time of our audit, the Ministry did not have the following information for 
monitoring and analysis purposes: 


¢ the number of inspections that resulted in inspectors’ orders being issued (orders list 
instances of non-compliance noted during inspections and direct the corrective action 
that needs to be taken), versus the number of inspections that demonstrated 
compliance; 


e the nature and seriousness of violations; 


¢ the number of outstanding inspectors’ orders and the length of time such orders had 
been outstanding; and 


¢ the number of high-risk establishments and repeat offenders. 


The only information provided to us to demonstrate the Ministry’s monitoring efforts was 
the number of prosecutions and convictions. We noted that for the five fiscal years ended 
2001/02, the TSSA reported an average of 23 prosecutions and eight convictions per year 
for the seven sectors that it regulated. Not having monitored and analyzed the effectiveness 
of follow-up actions taken by the TSSA on violations identified during its inspections, the 
Ministry did not have adequate information to assess whether the extent of prosecutions and 
convictions was appropriate and sufficient to deter future violations. 


As indicated in the Audit Objectives and Scope section of this report, our audit was not to 
assess the effectiveness of the various delegated authorities but to assess the adequacy of the 
procedures and systems the Ministry has in place to monitor these authorities and thereby 
ensure public safety and consumer protection. In that regard, the following 
recommendation applies to the Ministry’s monitoring efforts for all of the delegated 
authorities. 


Recommendation 


To better protect public safety the Ministry should improve its monitoring of 


delegated administrative authorities (delegated authorities) by: 


e ensuring the reliability of the outcomes that are reported by the delegated 
authorities; and 
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¢ monitoring the activities of the delegated authorities to ensure that 
inspections, investigations, enforcement measures, and other appropriate 
actions are taken on a timely basis and are sufficient to ensure the 
achievement of established safety outcomes. 


Ministry Response 


The Ministry recognizes the importance of reliable safety data and supports 
the delegated authorities in their efforts to improve the quality of this 
information. 


However, revisions to historic data have never hindered the ability of the 
government or the Technical Standards and Safety Authority (TSSA) to take 
timely and appropriate action to ensure public safety. Any individual safety 
incident that reveals a gap in safety standards or design requirements is acted 
on without delay. 


For example, with the passage of the Technical Standards and Safety Act and 
related regulations in 2000, Ontarians have benefited from a more modern and 
responsive legal framework that enables timely updates to safety standards 
and better regulatory tools to respond effectively to emerging safety issues. 
The Ministry and TSSA have implemented a number of other important 
enhancements to safety standards in recent years. Some examples of effective 
responses include: 


e one fatality on a bungee ride that resulted in approvals no longer being 
granted for non-stationary bungee devices in the Province of Ontario; 

e the decision to introduce certification requirements for amusement device 
and ski lift mechanics in response to recent incidents; and 

e arecent TSSA Director’s Order outlining clear safety requirements for tube 
tow device operators in response to two safety incidents last winter. 


The Ministry will continue to work with the safety authorities to ensure 
continuous improvement in the quality of their data and their approach to risk 
management. 


As well, the Ministry remains committed to requiring quarterly performance 
tracking reports from the safety authorities to ensure they can demonstrate 
appropriate and timely enforcement responses to safety incidents. 


Electrical Safety Authority 


In April 1999, under the Electricity Act, 1998, the Electrical Safety Authority (ESA) 
(formerly the Ontario Hydro Electrical Inspection division) was delegated the authority to 
oversee electrical safety. In fulfilling this responsibility, the ESA’s activities include inspection, 
investigation, enforcement, and other electrical safety services in Ontario. The ESA’s primary 


responsibility is interpreting and enforcing the Ontario Electrical Safety Code. 
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The ESA reports to the Ministry the key outcomes of its activities as a measure of its 
effectiveness in protecting public safety and consumers. Key indicators include: the number 
of reported incidents (for instance, power-line contact); the number of critical injuries 
(defined as loss of consciousness, injury to major limbs, or large amount of blood lost); the 
number of fatalities; and the number of complaints received. 


Both the Ministry and the ESA were concerned about the high number of electricians 
killed and the number of critical injuries on the job. Critical injuries and fatalities reported 
on from 1998 through 2001 are illustrated in the following graph. 


ESA's Reported Critical Injuries and Fatalities Among Electricians, 1998-2001 


50 7 - 
© Fatalities 
™ Critical Injuries 





























1998 1999 2000 2001 


Source of data: Ministry of Consumer and Business Services 


The Ministry and the ESA also expressed the following concerns: 


¢ While electricians must attend college and apprenticeship programs to receive their 
certificates of qualification, no requirement exists for them to update their training in 


electrical safety requirements and their knowledge of the Electrical Safety Code. 

¢ Licensing requirements for electrical contractors varied from municipality to 
municipality. 

Based on the above concerns, the ESA proposed that it be delegated the authority to 


become the registrar for the certification and licensing of electricians and to enforce 
requirements for the certificates of qualification. 


Recommendation 


* 


To help reduce electrical incidents involving serious injuries and the deaths of 
electricians, the Ministry and the Electrical Safety Authority should work with 
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other stakeholders to develop consistent safety standards for the training and 
initial and ongoing licensing of electricians working in Ontario. 


Ministry Response 


The Ministry and the Electrical Safety Authority (ESA) recognize that having 


properly trained and accredited electricians is an important component of 
electrical safety in Ontario. While the ESA has previously been responsible for 
electrical permits and inspection, the Ministry and the ESA have been actively 
working with industry groups, consumers, and the Ministry of Training, 
Colleges and Universities to strengthen the oversight and professional 
development and standards requirements for electricians. 





Ontario Motor Vehicle Industry Council 


The Ontario Motor Vehicle Industry Council (OMVIC) received delegated responsibility 
for the administration of the Motor Vehicle Dealers Act in January 1997. OMVIC is 


responsible for registering and monitoring motor vehicle dealers and salespersons, and 
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undertaking enforcement action with a focus on on-site visits and inspections, investigations, 
and prosecutions of those violating the Act. They also handle complaints and mediation, 
which provides alternatives to litigation for consumers and dealers. In addition, they 
promote professional standards and public awareness, which involves developing programs 
intended to enhance the industry’s professionalism and consumer knowledge. 


Key indicators of the OMVIC’s activities and outcomes include: dollars returned to 
consumers, the number of convictions resulting from investigations, the number of inquiries 
handled, and the number of inspections conducted. The OMVIC was the only delegated 
authority that provided the Ministry with up-to-date quarterly statistics on its activities and 
outcomes. Our review of the outcomes reported over the past five years indicated the 
OMVIC had consistently improved its effectiveness in having money returned to 
consumers over complaints relating to motor vehicle dealers. 


OMVIC’s Reported Activities and Outcomes, 1998-2002 


% increase 
Since 1998 


1998 1999 2000 2001 2002 








Activities 
Inquiries and complaints 


Inspections 
Outcomes — so ”,h,r,rt—~—~sSiCsCN 
PUBDES laid 1,457 1,774 ral 162 373 


ens | sar | — al — no 


Sources of data: OMVIC, Ministry of Consumer and Business Services 
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As shown in the above table, the money returned to consumers as a result of mediations and 
other enforcement activities in 2002 was $1.2 million, an increase of 100% from about 
$600,000 in 1998. Our review of the above statistics indicated that the significant 
improvement in OMVIC’s outcomes, such as the number of convictions and the amount of 
money returned to consumers, corresponded with the increase in its regulatory activities, 
such as the handling of complaints, the initiation of mediations and inspections, and the 


laying of charges. 
While the OMVIC is responsible for regulating motor vehicle dealers, the Ministry 


continues to be responsible for complaints related to motor vehicle repairs and related 
inspections and investigations. We noted that although motor vehicle repair was in the top 
three of complaints received by the Ministry over the past three years, the number of 
inspections the Ministry conducted was minimal—as shown in the following table. 


Ministry Data on Complaints and Inspections 
for Motor Vehicle Repairs, 2000-2002 


Written LAE 
Total 
Ministry in erections 





Source of data: Ministry of Consumer and Business Services 


Our review indicated that, because most dealers also conduct repairs on their premises, 
there was overlap in the functions of the Ministry and the OMVIC in terms of handling 


complaints and conducting inspections related to the industry. 











Recommendation 


To protect consumer interests with respect to the regulation of the motor 
vehicle industry in a more cost-effective manner, the Ministry should work with 
the Ontario Motor Vehicle Industry Council on ways to improve the 
effectiveness of consumer protection with respect to motor vehicle repairs. 


Ministry Response 


The Ministry has initiated discussions with the Ontario Motor Vehicle Industry 
Council to explore co-operative relationships to improve regulatory 
effectiveness in the motor vehicle repair sector. . 











114 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





Ontario New Home Warranty Program 


The Ontario New Home Warranty Program (ONHWP) was established in 1977 to 
administer the Ontario New Home Warranties Plan Act. The ONHWP was established to 
make new home warranty coverage mandatory. As a result, builders in Ontario are required 
to register as members of the ONHWP and pay enrolment fees for new home sales to cover 
potential warranty claims from new homebuyers in Ontario. Homebuyers can 
communicate complaints and submit them to the ONHWP if they are dissatisfied with the 
services provided by the builder. The ONHWP processes the complaint to determine if it is 
eligible for coverage. If it is, the ONHWP can compel the builders to make repairs or can 
pay out a suitable compensation to the homebuyer and may recover the refund from the 


builder. 


At the time of our audit, the Ministry had not been successful in negotiating an 
administrative agreement with the ONHWP to define their respective roles and 
responsibilities in protecting homebuyers in Ontario. We noted that because it did not have 
a formal agreement in place, the Ministry could not effectively provide any direction to the 
ONHWPB. or obtain adequate information about its operations in protecting homebuyers. 
For instance, the Ministry could only provide us with the ONHWP’s annual reports 
because it had limited success in obtaining additional information from the ONHWP. 


We compiled and reviewed information on performance outcomes that was available in 
public, ONHWP annual reports and other sources. We noted that there were no data on 
complaints to the ONHWP beyond 2000 because the ONHWP had stopped publishing 


complaints in its annual report. The following table outlines our compiled statistics. 


ONHWP’s Outcome Statistics, 1998-2000 


% Increase/ 





Outcomes : 1998 - 1999 2000 Decrease 


Since 1998 

Complaints against builders 21,200 22,900 28,153 33 
Homebuyers’ claims approved 1,100 -31 
Money paid out for claims ($ 000) 8,754 5.203 7,391 -16 


Sources of data: ONHWP and the Ministry of Consumer and Business Services 


















We noted that while new home sales in Ontario had steadily increased since 1998 and 
complaints had increased slightly in proportion, the number of claims approved had 
declined significantly and the amount paid out to consumers had declined as well. No 
information was available to explain the cause of the decline in claims approved. There were 
several possible causes for the decline: for instance, the ONHWP might be able to more 
effectively mediate complaints against builders, or it might be favouring builders in rejecting 


consumers’ claims. 
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Our review of complaints received by the Ministry specifically about the ONHWP showed 
that the complaints mostly related to the following: 


e that the ONHWP’s services were biased in favour of builders over consumers; and 


¢ that the ONHWP’s decisions were biased against smaller builders because 


representatives of big builders dominated its board. 


The Ministry indicated that it had expressed concerns regarding the above complaints and 
had met on numerous occasions with the ONHWP to discuss the complaints as well as a 
governance and accountability mechanism. 


Recommendation 


The Ministry should take action to ensure that better accountability 
mechanisms are in place to protect consumers buying new homes in Ontario. 


Ministry Response 


The Ontario New Home Warranty Program and the Minister of Consumer and 
Business Services have signed a letter of accountability establishing formal 
reporting requirements and outlining the roles and responsibilities of each 


party. 





Governance and Accountability of Delegated 
Authorities 


Delegated authorities were established to be more responsive to the needs of their industries 
and other stakeholders in the marketplace and more responsive in deploying resources for 
regulatory activities. To counter potential criticism of this self-regulatory model—specifically, 
criticism of under-regulation, under-representation of consumers, and biased decisions 
favouring the industries’ members—the Ministry put in place a number of formal and 
informal tools to ensure that delegated authorities remain accountable and fair and provide 
for sufficient consumer representation. These tools include: 


* administrative agreements signed between the Ministry and each delegated authority, 
which outline the roles and responsibilities of each party; 


¢ the Ministry's right to recommend or appoint government, public, and consumer 
representatives to the boards of directors of the various delegated authorities; and 


¢ the establishment of a ministry unit to monitor delegated authorities—specifically, their 
effectiveness outcomes and compliance with legislative requirements, including timely 
submission of annual reports, business plans, and other operational reports required by 


the Ministry. 
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We made several observations relating to these governance and accountability tools, which 
are outlined in the following sections. 


ADMINISTRATIVE AGREEMENTS 


The Ministry sets expectations and terms of delegation with the delegated authorities 
through administrative agreements. The agreements are an important tool in governing 
delegated authorities and making them accountable. The requirements outlined in these 
agreements include: the composition of the board of directors, annual performance 
reporting expectations, and oversight fees. We noted that the Ministry had not negotiated 
an administrative agreement with two of the eight delegated authorities. 


MINISTER’S APPOINTEES 


The Safety and Consumer Statutes Administration Act, 1996 enables the Minister to appoint 
up to 50% of the members of delegated authorities’ boards of directors. The presence of 
Minister's appointees on a delegated authority’s board helps maintain a balance between 
representation by government, consumers, the public, and the industry. At the time of our 
audit, appointments to the boards of the delegated authorities we reviewed were as outlined 


in the following table. 


Composition of Selected Delegated Authorities’ Board of Directors, Spring 2003 





* Because no administrative agreement was in place for the ONHWP, no provision existed for Minister's 
appointees on its board of directors. 


Sources of data: TSSA, ESA, OMVIC, ONHWP, 


and the Ministry of Consumer and Business Services 


Given that the Minister can appoint 50% of board members, we noted the significant 
under-representation of members who were independent of the industries being regulated 
in the composition of boards. Although board members might not belong to the industries 
being regulated, if they are appointed by the regulated industries, they are not fully 
independent. Also, we noted that attendance by ministry staff appointed by the Minister 
was generally below that of other board members. 


MONITORING OF AUTHORITIES’ PERFORMANCE 


Prior to the delegation of authority to the various delegated authorities, the costs of 
regulating the related industries were recovered by the Ministry through revenues that were 
generated through registrations, inspections, and certification activities. Since delegation, 
these revenues are collected by the delegated authorities. Revenues are to be used to cover 
the costs of the regulatory activities undertaken by delegated authorities as well as the 
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oversight function the Ministry has for the delegated authorities. To ensure the Ministry has 
sufficient resources to fulfill this oversight role, administrative agreements between the 
delegated authorities and the Ministry provide for the Ministry to recover some of the 
revenues collected by delegated authorities to cover its costs. 


When the delegation of authority began with the establishment of the first four delegated 
authorities in 1997, the Ministry’s original plan was to collect from delegated authorities 

$2 million in oversight fees per year to cover the costs of monitoring the performance of 
these delegated authorities. However, of the $2 million actually collected by the 
government, only $380,000 was allocated to the Ministry to provide four staff for this 
monitoring. Since 1999, four more delegated authorities have been added without 
additional funding. According to the Ministry, current staff levels were not sufficient to deal 
with the significant increase in workload. Our review indicated that this lack of resources 
could have contributed to the inadequate monitoring efforts already noted in this report. 


With respect to reporting requirements, the Safety and Consumer Statutes Administration 
Act, 1996 requires that delegated authorities submit annual reports showing their activities 
and financial affairs within four months of their fiscal year-end. This provision is intended to 
allow the Ministry to review and table the reports in the Legislature on a timely basis. Our 
review indicated that none of the delegated authorities complied with this reporting 
requirement and that the Ministry took from 12 to 20 months after the delegated 
authorities’ fiscal year-end to table their annual reports in the Legislature. 


Recommendation 


To better protect consumers and the public, the Ministry should strengthen its 
governance and accountability arrangements with delegated administrative 
authorities (delegated authorities) by: 


e establishing administrative agreements with the delegated authorities on a 
timely basis; 


having an adequate number of government, consumer, and public 
representatives on the boards of directors of the delegated authorities to 
achieve a fair balance of representation; 


ensuring sufficient levels of resources are devoted to monitoring the 
performance of the delegated authorities; and 


ensuring reporting and other performance requirements are complied with 
on a timely basis. 





Ministry Response 


With respect to establishing administrating agreements, five delegated 
authorities under the governing Safety and Consumer Statutes Administration 
Act, 1996 framework have administrative agreements in place. The Ministry has 
also undertaken a review of these administrative agreements and has 
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accountability best practices. 


In addition, the Ontario New Home Warranty Program and the Minister of 
Consumer and Business Services have signed a letter of accountability. 


Furthermore, as a result of the recently passed Funeral, Burial and Cremation 
Services Act, 2002, the Ministry has a legislative basis to work with the 
government and stakeholders to create a new cemetery and funeral service 
administrative authority. The Ministry will review governance and 
accountability provisions in the context of implementing the new Act. 


With respect to representation on boards of directors, the Ministry is 
committed to ensuring a fair balance of representation on the boards of its 
delegated authorities. The board of each delegated authority contains a 
substantial proportion of members who are independent from the regulated 
industries. The Technical Standards and Safety Authority (TSSA), for example, 
has adjusted its board balance to include nine non-industry directors and eight 
industry directors from distinctly different sectors that TSSA regulates. 


The Minister of Consumer and Business Services retains the authority to 
adjust the current board balance by appointing up to 50% of the board. The 
Ministry will continue to review the composition of the delegated authority 
boards over time to ensure that an appropriate proportion of non-industry, 
independent directors is maintained. 


Furthermore, the Ministry will develop a skills profile reflecting the ideal 
competencies sought for non-industry members, in order to optimize non- 
industry member effectiveness on these boards of directors. 


With respect to ensuring sufficient resources for performance monitoring, the 
Ministry will review its capacity to fulfill oversight functions and will make 
necessary staffing adjustments where warranted. 


Finally, with respect to ensuring timely compliance with performance 
requirements, the Ministry will develop a tracking process for the tabling of 
delegated authorities’ annual reports in order to better monitor the movement 
of reports through the process within the Ministry and to take timely corrective 
action when necessary and within the Ministry’s authority. 


The Ministry will also improve its tracking of delegated authorities’ 
performance statistics to ensure timely quarterly reporting to the Ministry on 
relevant performance measures. 
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MINISTRY OF EDUCATION 


3.05—Curriculum 
Development and 
Implementation 





BACKGROUND 


The Education Act gives the Minister of Education broad authority over the “courses of 
study that shall be taught” to the province’s 1.4 million elementary and 700,000 secondary 
students in its 4,000 elementary and 800 secondary schools. The province’s 72 school 
boards are responsible for ensuring that their staffs comply with provincial policy on 
education and for helping teachers to improve their teaching practices and to deliver the 
curriculum effectively. School boards, through their trustees, are also accountable to the 


local electorate for the quality of education provided. 


The process of reforming elementary and secondary education started with the creation of 
the Royal Commission on Learning by an Order-in-Council in May 1993. In response to 
the recommendations of the Royal Commission in December 1994, to the 
recommendations in our 1993 Annual Report, and to public concerns about the quality of 
education, the Ministry of Education assumed full responsibility for curriculum policy and 
in 1996 undertook, for the first time, the development of a province-wide curriculum. The 
new elementary curriculum was introduced concurrently for all grades in September 1997 
and 1998, while the new four-year secondary school curriculum was introduced one grade 
at a time, starting with the grade 9 curriculum in September 1999 and ending with the 
grade 12 curriculum in September 2002. The first students to graduate from secondary 
school after having been taught the new curriculum since grade 1 will be the 2009/10 
cohort. 


The Ministry estimated the costs of developing the new curriculum policy documents to be 
about $16 million between 1996 and 2000. It estimated that total implementation costs 
incurred between 1997 and January 31, 2003, were about $472 million, as outlined in the 
following table. 
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Estimated Costs of Implementing the 
New Curriculum from 1997 to January 31, 2003 


Pe uiiiceya)) 


F 1 
New textbooks and learning resources 


Training for educators 
Teacher resource materials 
New report card 

Ministry staffing costs 





" Includes $14 million from the federal government for French-language 
education. 


Source of data: Ministry of Education 


Responsibility for curriculum policy and education programs rests primarily with the 
Curriculum and Assessment Policy Branch of the Strategic Planning and Elementary/ 
Secondary Programs Division, and the French Language Policy and Programs Branch of the 
French-Language Education and Educational Operations Division, as set out in the 
following table. 


Curriculum Responsibility and Resources 


i Te 2002/03 
Branch Stat 


Curriculum and Assessment Policy 
French-Language Policy and Programs ia ee hove: 


* Over 90% of these expenditures are for grants and service contract payments made to school 
boards, educational associations, and publishers for curriculum- and learning-resource-related 
projects. 





Source of data: Ministry of Education 


In response to a recommendation by the Royal Commission, the Education Quality and 
Accountability Office (EQAO) was established as a Crown agency in 1996. The EQAO’s 
Board of Directors reports to, and operates under a Memorandum of Understanding with, 
the Minister of Education. Its main responsibilities are to evaluate and publicly report on 
the quality and effectiveness of elementary and secondary school education in Ontario and 
on the public accountability of the province's school boards. The EQAO had expenditures 
of $50.7 million for the year ended March 31, 2003. 
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AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit were to assess the adequacy of the Ministry’s procedures for: 


° ensuring that its curriculum meets the province’s objectives for the education of students 
and that it is implemented cost effectively; and 


¢ measuring and reporting on the effectiveness of both curriculum content and 
implementation and, where necessary, ensuring that appropriate corrective action is 


taken. 


Our audit focused primarily on the related activities of the Ministry’s Curriculum and 
Assessment Policy Branch and the French Language Policy and Programs Branch. We also 
interviewed EQAO personnel to gain an understanding of their procedures for conducting 
and reporting on the results of province-wide testing as well as other activities they have 
undertaken to fulfill their mandate. In addition, we interviewed administrative personnel 
and reviewed policy and other documentation at three English-language school boards and 
one French-language school board and interviewed a sample of principals and teachers, 
primarily from these school boards. 


We identified audit criteria that would be used to conclude on our audit objectives. These 


were discussed with and agreed to by senior management of the Ministry and the EQAO. 


Our audit was carried out from September 2002 to June 2003. The Ministry’s Internal 
Audit Services Branch had not done any recent work that allowed us to reduce the scope of 
our work, 


Our audit was conducted in accordance with professional standards for assurance 
engagements, encompassing value for money and compliance, established by the Canadian 
Institute of Chartered Accountants, and accordingly included such procedures as we 
considered necessary in the circumstances. 


OVERALL AUDIT CONCLUSIONS 


We concluded that the process by which the Ministry developed the new elementary and 
secondary curriculum was appropriate and found that, according to most of the educators 
we interviewed, it resulted in a good-quality product that was an improvement over what 
they had before. However, the process of implementing the new curriculum and related 
reforms is ongoing, and the Ministry and school boards did not yet have sufficient assurance 
that: 


¢ adequate procedures are in place to ensure that the 4,800 schools throughout the 
province are teaching the new curriculum, that students are being properly and 
consistently assessed, and that best practices have been implemented; and 


* there is sufficient appropriate information to assess the performance and management 
of schools with respect to student achievement. 
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Although we found general satisfaction with the content of the new curriculum, educators 
we interviewed expressed concerns regarding the way it was implemented. Their major 
concern was that the Ministry rushed the implementation with the result that a new 
curriculum and changes in student assessment practices were introduced before appropriate 
training, textbooks, and other materials were readily available. This made the initial years of 
implementation extremely difficult for students and teachers. 


Educators also expressed concerns about the suitability of the new curriculum for weaker 
students. Recent studies and test results indicated that many students are still not succeeding 
under the new curriculum and that revisions to the curriculum or teaching strategies or 
both are required to help these “at-risk” students succeed. 


While the Ministry and the school boards we visited have introduced initiatives to assist at- 
risk students, we found that the Ministry had not provided adequate guidance regarding 
the promotion of at-risk students. At-risk elementary students are promoted or transferred 
to the next grade without upgrading their knowledge and skills by participating in remedial 
programs. As a result, many students are entering secondary school without the educational 
foundation required to graduate. 


We also concluded that the Ministry and the school boards we visited did not have 
sufficient, reliable student performance and contextual data to compare and interpret 
student achievement results and consequently were not able to: 


¢ measure and report on the extent to which students have learned the new curriculum in 


grades and subjects other than those that are tested by the EQAO; 


¢ measure the extent to which consistency in student assessment has been achieved among 


the province’s 4,000 elementary and 800 secondary schools; 


¢ identify and prioritize the problems underlying poor student achievement at under- 
performing schools, to develop viable improvement plans, and to measure and report 
on the extent to which the steps taken to improve student performance had been 
successful; and 


¢ conduct the research necessary to address critical issues in curriculum delivery and 


provide the basis for informed decision-making. 


The Ministry funded the development of course profiles and an electronic lesson planner to 
enable teachers to prepare lesson plans more efficiently and effectively. However, many 
teachers we interviewed did not consider these tools to be useful to them. Superintendents 
felt that a lack of computer literacy and access to computers was an impediment to increased 
use of these tools. 
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DETAILED AUDIT OBSERVATIONS 
CURRICULUM DEVELOPMENT 


Under the ministry policies and guidelines in place prior to the introduction of the new 
curriculum, school boards had considerable latitude regarding the curriculum that was 
taught, with the result that there was little consistency across the province. For example, for 
grades 10 to 13, the Ministry found that there were approximately 700 different courses 
across the province. (These have since been reduced to 184 standard secondary school 
courses.) 


Coincident with the development of a new curriculum, the number of years of elementary 
and secondary schooling were reduced from 13 to 12, which is consistent with most other 
jurisdictions, and which was recommended by the Royal Commission on Learning in its 


1994 report. 


In establishing its new curriculum standards, the Ministry reviewed the curriculums of a 
number of other jurisdictions and took advice from, among others, a number of Ontario 
school boards, various subject associations (for example, the Ontario Association for 
Mathematics Education), and specialists in designated subject areas. In addition, for the 
secondary school curriculum, the Ministry established a validation process in which 
representatives from universities, colleges, business groups, the professions, and the trades 
assessed and ultimately approved the appropriateness of the learning expectations for post- 
graduation destinations: university, college, and the trades/workplace. 


Structure of the New Curriculum 


The Ministry’s curriculum development process resulted in the publication of 41 
curriculum policy documents in both English and French: 10 for elementary (including the 
Kindergarten Program) and 31 for secondary. The policy documents are organized into 
strands, which are broad areas of study or focus for each subject area in each grade (at the 
elementary level) and each course (at the secondary level). Each strand contains both overall 
and specific learning expectations describing the skills and knowledge the student is 
expected to acquire. 


Each policy document also contains an achievement chart for each subject (at the 
elementary level) and for each discipline (at the secondary level). These charts are the 
framework used for assessment and evaluation of student achievement. They describe four 
different levels of achievement of the curriculum expectations. Level 3 (the second-highest 
level) is the provincial standard and identifies a high level of achievement of the provincial 
expectations. 


The basic structure of the new elementary and secondary curriculums is set out in the 
following chart. 
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Structure of the Elementary/Secondary Curriculum 
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Source of data: Ministry of Education 


In addition to the types of secondary school courses listed in the chart above, boards may 
offer: 


¢ locally developed courses, with prior ministry approval, to serve local needs or interests, 
or the needs of special education students; and 


¢ transfer courses to enable students who change their destination plans to upgrade their 
skills in a particular subject (for example, to change from Applied Math to Academic 
Math). 


Students are expected to select their courses according to their ultimate destination. Thus, a 
student intending to study English at university would take University Preparation courses 

in English, but might take University/College or College Preparation courses in 
Mathematics. Academic courses in grades 9 and 10 serve as prerequisites for all four levels of 
grade 11 and 12 courses, whereas Applied courses do not satisfy prerequisite requirements 
for university preparation courses. Consequently, the majority of students choose academic 
courses in grades 9 and 10. 


Views on Curriculum Content 


The teachers and principals we interviewed supported the idea of having a province-wide 
curriculum. Almost all teachers we interviewed considered the new curriculum to be an 
improvement over what they had before, with more than 40% rating it as a major 
improvement. However, concerns were expressed that the new curriculum might not meet 
the needs of weaker students. 


All the principals we interviewed believed that the new curriculum was as good or better for 
most students than what they had before. However, we received mixed views from 
principals on the suitability of the curriculum for weaker students at the elementary level 
and for the estimated one-third of secondary school graduates who join the workforce 
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directly. Several secondary principals we interviewed were also concerned that the work 
habits of their work-bound students had actually deteriorated since its introduction. 


Conclusion on Curriculum Development Process 
The process by which the Ministry developed the new elementary and secondary 


curriculums was designed to ensure that provincial objectives, such as establishing a 
challenging, rigorous curriculum with high provincial standards, would be met; appropriate 
expertise and stakeholder involvement was employed in its development; and the 
curriculums of jurisdictions that had recently undertaken curriculum reform were used as 
benchmarks. Its appropriateness for weaker students is discussed later in this report. 


IMPLEMENTATION OF THE CURRICULUM 


With respect to curriculum implementation, school boards are responsible for ensuring that 
their staffs comply with provincial policy on education and for helping teachers to improve 
their teaching practices and to deliver the curriculum effectively. Principals are responsible 
for supervising and evaluating the performance of teachers in providing the appropriate 
instruction for their students and in evaluating student work and progress. Procedures for 
ensuring that school boards and their staffs comply with legal and policy requirements and 


perform their roles effectively are discussed later in this report. 


Implementation Problems 


The general satisfaction that educators had with the content of the curriculum did not 
apply to the way in which it was implemented. The primary concern was that the Ministry 
rushed the implementation with the result that: 


¢ ‘Teacher training on the new curriculum and on use of the achievement charts for 
evaluating student work was not conducted early enough or, in some cases, at all. The 
Ministry's “train the trainer program” was not viewed as a success by most educators we 
interviewed, primarily because insufficient release time (that is, time off during the 
instruction day) was set aside to implement this program at the schools. The Ministry 
stated that, in addition to the two days of release time per teacher for training on the 
new curriculum that it funded at a cost of $36 million, it expected teachers to do some 


of the required training on their own time. 


In this regard, the Ministry noted that, between 1999 and 2001, approximately 
22,000 teachers obtained training in the implementation of the new curriculum on 
their own time at the Summer Institutes. (The Institutes are workshops managed by the 
Ontario Teachers Federation and funded by the Ministry. They were initially established 
to assist teachers in upgrading their skills to implement the more rigorous curriculum. 
They have evolved into a more general professional-development vehicle for teachers 
and include topics such as classroom management in addition to subject-specific 
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courses.) Principals and superintendents stated that, because some of their teachers were 
either reluctant or unable to attend courses after hours during the school year or during 
the summer, the number of professional development (PD) days should be increased 
from four to at least eight per year. 


Appropriate textbooks and classroom materials were in many cases not available when 
classes commenced, making the initial years of implementation extremely difficult for 
students and teachers. However, the majority of teachers we interviewed indicated that 
this problem is being resolved over time as publishers introduce new textbooks and 
supporting materials. Suitable textbooks now exist for core English-language elementary 
and secondary subjects and courses. Availability remains an issue, however, for French- 
language grades 7 to 12—there are still a number of courses for which the Ministry has 
not yet approved a textbook—and for non-core courses such as visual arts in both 


languages. 


Focus of Teacher Training 


The primary ongoing concern of both teachers and principals whom we interviewed was 
training. The two most common suggestions made by teachers we interviewed regarding 


how training could be improved were to: 


offer more grade-level courses open to teachers from a number of schools, in order to 
provide opportunities to discuss successful instructional strategies and techniques with a 
wider range of colleagues; and 


(particularly emphasized by elementary teachers) orient courses more to the actual 
implementation of strategies and techniques rather than to the largely informational 
sessions that they had attended so far. 


Use of Teacher Supports 


The introduction of a province-wide curriculum to be used by all teachers, versus the 
variety of curriculums previously in use, made it practical for the Ministry to assist teachers 
in the delivery of the curriculum by funding the development of tools for common use, 
such as: 





course profiles, which were created, at a cost of $33 million, by boards and subject 
associations to assist secondary teachers in delivering each course. The profiles build 
upon the specific learning expectations set out in the policy document for each course; 
provide teachers with a suggested sequence for presenting each concept; and specify the 
amount of time that should be devoted to each strand. They also include suggested 
teaching strategies and a list of resources that the teachers may want to consider using 
during the course. The profiles are available to teachers in electronic format so that they 
can customize and incorporate them into their lesson plans. 
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° — the curriculum unit planner, a software tool that cost $3 million to develop and is 
intended to enable both elementary and secondary teachers to efficiently develop lesson 
plans for each subject or course. 


Despite the significant efforts by leading educators that went into developing the course 
profiles, only one-third of the secondary teachers who responded to our questionnaire rated 
the profiles as adequate in respect to usefulness. This result is consistent with the finding in 
the Ministry’s June 2001 Mathematics Survey that teachers had not “used the course 
profiles to any extent.” Only about 60% of the elementary and secondary teachers that we 
interviewed rated the unit planner as useful. Superintendents we interviewed thought that 
insufficient computer literacy was an impediment to greater use of these and other software 
tools. As well, teacher access to computers is an issue because school boards we visited did 
not provide teachers with PCs. We were told by the superintendents we spoke with that 
doing so is an objective but not a high priority one. 





Recommendation 


To help ensure that future revisions to the curriculum are implemented more 
effectively, the Ministry should ensure that: 





e teachers receive appropriate training prior to implementation; and 


e educational publishers have sufficient lead time to develop appropriate 
textbooks and classroom materials. 


To help improve the implementation of the current curriculum, the Ministry 
should work with school boards to ensure that teachers receive more specific 
implementation training, including training on the use > of tools such as the 
course profiles and unit planner. 


Ministry Response 


Teacher training and classroom resources are important components of an 
effective implementation plan. 


The Ministry is pleased that the audit report recognizes that the process for 
developing the new curriculum was appropriate and that most educators the 
auditor interviewed consider the curriculum to be a high-quality product. 


Over $300 million was spent on textbooks and learning resources to support 
the new curriculum from kindergarten to grade 12 in both English and French. 
As a result of ongoing textbook development, an increasing number of grade 
11 and 12 textbooks have since been made available for French-language core 
and non-core courses for the 2003/04 school year. If new funds were to be 
allocated to learning resources, ensuring the availability of appropriate 
textbooks for additional subjects and courses will be considered. 


The Ministry is implementing a multi-year plan to sustain a quality curriculum. 


As part of our plan, we will continue to work with our educational partners to 
Lo oe ee) 











128 2003 Annual Report of the Office of the Provincial Auditor of Ontario 


ensure that teachers receive appropriate and timely training to implement 
future curriculum change. We will also work with our partners to identify areas 


for additional focused and practical training and with publishers and other 
interested stakeholders so they have the necessary lead time to develop 
appropriate resources to support curricular revision. 





ADDRESSING THE NEEDS OF AT-RISK 
STUDENTS 


One issue that teachers and principals raised regarding the content of the new curriculum 
was its suitability for weaker students. Educators stated that both the elementary and 
secondary curriculums cover more material and are more challenging than the old 
curriculums. This leaves less classroom time for students to master each concept, and as a 
result, weaker students may fall behind. By the time they enter secondary school, these 
students are at risk of being part of the 25% of students noted in the Ministry's October 
2002 “Double Cohort Study—Phase 2 Report” who leave school without obtaining an 
Ontario Secondary School Diploma. 


Promotion without Remediation 


We found that there was strong agreement among the teachers we interviewed that students 
need to perform at least at level 2 (60-69%) in a subject in order to have a sufficient 
understanding of the key learning expectations to succeed in subsequent grades. The 
Ministry, however, has set the pass rate at level 1 (50-59%), with the result that students at 
this performance level are promoted to the next grade without any requirement that they 
take steps to raise their understanding of key concepts to at least level 2 by attending 
summer school or doing other remedial work. 


Under the Education Act, promotion decisions are made primarily by principals. We were 
told by the elementary school principals we interviewed that even students who perform 
below level 1 are normally transferred to the next grade. This practice is known as “social 
promotion.” Although teachers and principals might recommend that these students 
upgrade their knowledge and skills through summer school or other remedial programs, 
they do not require them to do so as a condition of being promoted. The Ministry stated in 
its 2001/02 business plan that it intended to require that schools promote only those 
students who achieved at an acceptable level, that is, one that would ensure that they had 
the skills and knowledge needed to progress through the education system. However, no 
action was taken to implement this intention. 


None of the boards we visited tracked and monitored the extent to which students who 
achieved below level 2 participated in remedial programs or the impact of such programs 
on students’ subsequent performance. One administrator who co-ordinated all the remedial 
programs at one of the boards stated that the proportion of at-risk students who took 
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advantage of the board’s remedial programs was quite low, and that it varied significantly 
among the board’s schools. Low participation in remedial programs may be a major reason 
why at-risk students fail to acquire the knowledge and skills necessary to succeed at 
secondary school. 


In the United States, the Chicago Public Schools (CPS), which ended social promotion in 
the 1996/97 school year, reported in 1999 that the percentage of grade 3, 6, and 8 
students who had marks good enough to pass to the next grade increased to 82% from 
76% in 1997. The chief executive officer of CPS stated that the improvement would not 
have happened without mandatory summer school for failing students coupled with the 
threat of being retained. Although other jurisdictions are experimenting with more 
stringent promotion policies, such as the mandatory summer school programs at the CPS, 
the Ministry has not introduced any policy directive pertaining to promotion decisions or 
remedial programs. 


The results on the 2002 Ontario Secondary School Literacy Test (OSSLT) and the grade 9 
math test confirm that many students have not acquired sufficient literacy and math skills. 
The OSSLT is designed to test only basic literacy skills, not students’ comprehension of the 
secondary school curriculum. Yet 28% of first-time and 52% of previously eligible writers 
were unable to pass this basic skills test after nine and 10 years of schooling respectively. 
Students taking mostly Applied courses performed particularly poorly on the OSSLUT, with a 
62% failure rate among first-time eligible students. 


Ontario Secondary School Literacy Test 


First-time eligible students 


Previously eligible students 


Required to pass reading only 





Source of data: EQAO 


As shown in the table that follows, students taking Applied Mathematics also performed 
poorly on the 2002 EQAO grade 9 math test. Only 58% (37% level 2 + 21% level 3) were 
able to achieve the minimum level 2 performance that teachers feel is necessary to succeed 
in subsequent grades. 
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EQAO Grade 9 Math Test Results 
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The test results for students taking Applied courses confirm that many of them are at-risk 
students. Educators variously attributed the lack of success of these students to a curriculum 
that is too hard, poor work habits and low motivation, and ineffective instructional 
techniques. The primary concern regarding poor work habits was the failure to complete 
assignments, a finding that was also noted in the Ministry’s June 2001 Mathematics Survey. 
A study performed at one board, in connection with a doctoral thesis, also with respect to 
mathematics, found that secondary school teachers felt that social promotion in elementary 
school resulted in the affected students not having “the [necessary] background in Math” or 
the motivation to do the work required to succeed. In this regard, the teachers noted that 
some grade 9 students were unconcerned about their lack of math skills, “having the 
misperception that even if they did not understand the material, they would be promoted 
anyway, just like in elementary school.” 


With respect to the need to adopt different instructional techniques for students in applied 
courses, some of the secondary school math teachers interviewed in the above study noted 
that students in Applied Mathematics responded better to a different type of instructional 
method than that used in the traditional approach to teaching mathematics. Another study 
conducted by the same board to identify problems with the implementation of the new 
secondary school curriculum observed that “many teachers, especially new ones, have not 
the [instructional] strategies and tactics necessary to deal with students in Applied classes.” 


Ministry Initiatives Regarding At-risk Students 


The Ministry introduced an Early Reading Strategy in 2001 to help teachers to better assist 
students in acquiring a strong foundation in this core skill area in grades 1 to 3. It also plans 
to introduce a similar Early Math Strategy for grades 1 to 3 and to expand the Early 
Reading Strategy to grade 6 beginning with the 2003/04 school year. 


In November 2002 the Ministry assembled the At-Risk Working Group, composed of 
representatives from school boards and other major stakeholder groups, to provide the 
Ministry with recommendations to improve the performance of at-risk students. The 
recommendations in the Group’s January 2003 report included: developing an inventory of 
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successful school practices in the areas of literacy and numeracy, the application of 
diagnostic assessments to identify and track at-risk students, and teacher training and 
information sharing on assessment and instructional methodologies. 


In March 2003, the Ministry announced a $50 million program to address the Group’s 
recommendations. One of the objectives of this program and the Early Reading and Math 
Strategies is to address the problem of ineffective teaching strategies for at-risk students. 
However, these programs are not designed to address the deficiencies in student work habits 
that teachers believe are an important factor in poor performance at the secondary level. 


In addition, in February 2003, the Ministry began a systematic review and update of all 
curriculum policy documents from grades 1 to 12. This review will be completed over a 
five-year period, starting with the Social Studies/History and Geography elementary 
curriculum and the Canadian and World Studies secondary curriculum. The 
appropriateness of the curriculum for Applied students will be considered as part of this 
review. 





Recommendation 


To help ensure that the curriculum serves the needs of all students, the 
Ministry should: 


e develop policy guidance governing the promotion of at-risk students, 
including ways to increase participation in remedial programs such as 
summer school, to help ensure that all students acquire the knowledge, 
skills, and work habits required to succeed in subsequent grades and 
ultimately to obtain an Ontario Secondary School Diploma; and 

¢ require boards to track the participation of at-risk students in remedial 


programs and to assess the effectiveness of the programs in improving 
student performance. 





Ministry Response 


Addressing the learning requirements of students at risk of not succeeding is 
a ministry priority. We are pleased that the audit report recognizes the 
Ministry’s commitment to students who are at risk through initiatives related to 
the implementation of the recommendations of the At-Risk Working Group 
report, and the additional funding provided to support these initiatives. 





The Ministry will provide support to principals to ensure effective promotion 
practices for at-risk students, including the possibility of participation in a 
range of remedial programs. The Ministry will work with boards to develop 
processes to track student participation in remedial programs and to assess 
how effective the programs are in improving student performance. We will 
introduce appropriate accountability measures relating to the at-risk student 
funding provided to boards. The Ministry is also committed to implementing 
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the recommendations related to remediation put forward by the At-Risk Work 
Groups and Expert Panels. The Ministry will continue to review the research on 
the impact of social promotion and student retention. 


MONITORING CURRICULUM QUALITY AND 
IMPLEMENTATION 


One of the recommendations made by the Royal Commission on Learning was that an 
agency, independent of the Ministry, should construct, administer, mark, and report on 
large-scale assessment of student achievement. It was the Commission’s view that large-scale, 
system-wide testing was necessary “as a check on student learning at a few critical transition 
points, and as a vehicle for assuring people that, at those points, all students are being 
assessed according to the same yardstick.” 


Following this recommendation, the Education Quality and Accountability Office 
(EQAO) was established as a Crown agency in 1996. Its Board of Directors reports to, and 
must operate in compliance with, a Memorandum of Understanding (MOU) with the 
Minister of Education. The EQAO’s legislated mandate is: 


¢ to develop systems for evaluating the quality and effectiveness of elementary and 
secondary school education, including developing tests and administering and marking 
tests of pupils in elementary and secondary schools; 


* to research and collect information on assessing academic achievement; 


* to evaluate the public accountability of boards and collect information on strategies for 
improving that accountability; and 


* to report on and make recommendations regarding the effectiveness of elementary and 
secondary school education and on the public accountability of boards. 


The EQAO develops, administers, and marks two types of large-scale tests for the province's 
elementary and secondary students: 


¢ the Ontario Secondary School Literacy Test (OSSLT), which is normally written in 
grade 10 and is used to determine whether students have acquired the basic literacy 
skills as expected at the end of grade 9; and 


¢ grade 3 and 6 tests of reading, writing, and mathematics, and the grade 9 Academic 
and Applied mathematics tests, which are used to determine the extent to which 
students have achieved the learning expectations set out in the curriculum for these 
subject areas. 


The EQAO is also responsible for managing and reporting on the province's participation 
in national and international tests. These tests show how well Ontario students acquired 
certain core knowledge and skills compared with students in other jurisdictions. As a history 
of results subsequent to the implementation of the new curriculum is developed, these test 
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results will serve as a competitive benchmark against which the quality of the Ministry’s 
curriculum and its delivery by school boards can be assessed. 


Despite its comprehensive mandate, the EQAO’s activities have so far been limited to 
administering the aforementioned tests, reporting the results with some analysis, and 
providing guidance to school boards on improvement planning. 


Validating the EQAO’s Testing Practices 


In addition to using comparisons of the annual EQAO results of schools to identify 
potential problems, multi-year, large-scale test results are intended to enable the Ministry 
and boards to monitor the trend in student performance over time in order to determine 
whether student achievement is improving, staying about the same, or declining. 


‘Trends are meaningful, however, only if year-to-year results are comparable. Thus, the 
EQAO has procedures that are intended to ensure that year-to-year changes in the marks 
on its tests reflect changes in student performance rather than changes in the level of 
difficulty of the tests. The EQAO’s test development process includes field tests to obtain 
assurance that each set of tests is of similar difficulty; however, this step alone cannot 
guarantee that each year’s tests will be of the same level of difficulty. Consequently, the 
EQAO also uses a complex, statistically driven process known as “equating” to compensate 


for differences in test difficulty. 


At the time of our audit, the EQAO’s procedures for designing and reporting results on its 
large-scale tests had not been subjected to an independent examination by experts in the 
field of psychometrics. In view of the importance of the decisions that are based at least in 
part on EQAO results and the professional judgments required to design and administer 
complex testing processes, early in 2003 the EQAO engaged independent experts to review 
its processes. It is expected that the review will be completed by March 2004. 


Achieving Consistency in Student Assessment 


In connection with the introduction of a new curriculum, the Ministry implemented 
policies governing student assessment practices with the goals of: 


¢ improving student learning; 
* developing greater consistency in assessment and evaluation practices; and 
* providing clarity in reporting. 


Previously, boards had had considerable flexibility in their assessment practices. We were 
told at the schools we visited that practices had varied at the school level within boards and, 
at large schools, even within individual schools. During previous audits, we were told by 
admissions personnel at some universities that assessment practices varied so widely between 
secondary schools that it was necessary to develop processes for taking these differences into 
account 1n making admissions decisions. 
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To help meet its goals, the Ministry developed standard report cards and achievement charts 
that must be used by all schools. It also funded the development of: exemplars for both 
elementary and secondary subjects/courses; and an assessment and evaluation resource 
guide for secondary school teachers. The exemplars build on the achievement charts in the 
Ministry’s policy documents and provide teachers with examples of actual student work 
that the specialist teams have evaluated as representing level 1 (50-59%), 2 (60-69%), 

3 (70-79%), and 4 (80-100%) work. Because they use actual student work, the exemplars 
were not available until the year after the implementation of the curriculum and 
consequently will not be available for grade 12 courses until the 2003/04 school year. 
Exemplars were rated as being adequate or excellent by 82% of elementary and 71% of 
secondary teachers we interviewed. 


The boards we visited have also undertaken measures, such as providing training for 
teachers and principals in student assessment, developing and publishing guides on 
appropriate assessment practices, and encouraging principals to place higher priority on 
ensuring that their teachers follow appropriate student assessment practices. However, 
neither the Ministry nor the boards we visited had implemented procedures for monitoring 
and reporting on the extent to which consistency in student assessment has been achieved. 


We note that most of the principals and teachers we interviewed believed that consistency in 
assessment was improving within their individual schools. Particularly at the secondary level, 
though, they felt that board-wide consistency in student assessment had not been achieved. 


One possible method of measuring consistency in assessment for at least some grades and 
subjects is to compare report card marks to student performance on the grades 3, 6, and 9 
EQAO tests. Although effective classroom assessment will involve a wider variety of 
assessment approaches at more frequent intervals, the expectation would be that the final 
report card marks of most students in reading, writing, and math for grades 3 and 6, and in 
math for grade 9, would be comparable to their EQAO test marks. The superintendents we 
interviewed advised us that their boards planned to do this once the EQAO begins 
reporting student marks using the Ontario Education Number, the unique identifier for 
each student that the Ministry plans to introduce in September 2003. 


One way to obtain additional benchmark information for assessing consistency in student 
assessment in other grades and subjects would be to implement additional EQAO tests on 
all core subject areas at key points, such as grade 8 and 12 exit tests. For example, Alberta 
administers grade 12 diploma exams that count for 50% of students’ final marks in 12 core 
courses. Alternatively, province-wide exams could be used in place of class or school exams, 
particularly if accompanied by independent team marking. Such exams would thus provide 
a benchmark against which other classroom assessments could be compared. We note that 
the superintendents we spoke with could not describe any benefits to having more than 
4,000 different exams across the province for the same subject at the elementary level and 
800 or more exams for the same course at the secondary level, given that all exams are 
expected to assess achievement of the same learning expectations. 
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Gaps in Student Achievement Information 


As well as providing benchmark information for assessing consistency, the use of province- 
wide classroom exams or additional EQAO tests would help solve the problem of 
insufficient information about student achievement levels, particularly in secondary schools. 
As the Task Force on Effective Schools noted in its June 2001 report, “At the secondary level 
...we know almost nothing on a province-wide basis about achievement levels. The available 
information comes from teachers and from examinations set by individual schools, and very 
often by individual teachers. The new grade 9 math test and the grade 10 literacy test 
administered by the EQAO will tell us more about students’ elementary education than 
about their achievements at the secondary level.” As of 2003, this was still very much the 
case. 


Strengthening Implementation Processes 


The Ministry and school boards were still in the process of implementing the new 
curriculum and related reforms. We concluded that they did not yet have sufficient 
assurance that: 


* adequate procedures are in place to ensure that the 4,800 schools throughout the 
province are teaching the new curriculum, that students are being properly and 
consistently assessed, and that best practices have been implemented; and 


* appropriate accountability frameworks are in place—for example, the Ministry does not 
know whether trustees and school councils receive adequate information to assess the 
performance of their managements and schools with respect to curriculum 
implementation and student achievement, or whether if they do get such information, 


they use it properly. 


Such assurance could be obtained by conducting accreditation reviews such as those 
recommended by the Task Force on Effective Schools in its June 2001 report. The Task 
Force recommended that an agency at arm’s length from the Ministry design and 
implement an accreditation process for school boards and schools. The recommended scope 
of the accreditation examinations included: 


* the process used by boards to review their schools’ performance and curriculum 


delivery; 
* the adequacy and accuracy of information about student performance; 


* the process by which resources were allocated by the board to support student learning; 
and 


* the adequacy of board and school improvement plans. 


The scope of the first two items could be expanded to include student assessment practices 
and would thereby provide additional information on and more confidence regarding the 
consistency across the province. 


eee 
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If a system of province-wide subject and course exams accompanied by reviews of school 
assessment practices helped achieve the Ministry’s consistency objective, the Ministry would 
then be in a position to reconsider the need for large-scale testing by the EQAO. 


We also note that having comparable student performance results among the schools of 
each board, and especially among all schools within the province, would provide schools, 
school boards, and the Ministry with valuable information for identifying problems and best 
practices and would also help trustees and school councils hold their managements and 
principals accountable for results. 


The Ministry has not acted upon the recommendation by the Task Force. 


Measuring Outcomes 


While the Ministry publishes the province’s ranking on national and international tests as an 
indicator of its performance regarding curriculum quality, neither the Ministry nor the 
EQAO has developed outcome-oriented measures of effectiveness for the elementary/ 
secondary education system. Some possible measures include the percentage of students 
entering secondary school who graduate, the percentage of graduates who obtain degrees 
and diplomas from universities and colleges or who successfully complete apprenticeship 
programs for the trades, and employer satisfaction surveys for graduates who go directly into 
the workforce. 





Recommendation 


To help determine whether the Ministry’s expectations for curriculum reform 
are being met, and to enhance the public accountability of school boards, the 
Ministry should: 


e implement procedures to monitor and report on consistency in teachers’ 
student assessment practices throughout the province; 

e assess the benefits of developing common province-wide exams; 

e establish a process for strengthening school board implementation 
processes, the scope of which includes evaluating the adequacy of key 
curriculum delivery, student assessment, improvement planning, and 
results reporting procedures of school boards; and 

e develop and report on outcome-oriented measures of effectiveness for 
elementary and secondary education. 





Ministry Response 


Ensuring greater consistency in student assessment and evaluation and 
enhancing school board accountability are objectives which the Ministry 
supports. 





The Ministry has invested in strategies, resources, and teacher training to 
improve consistency in the assessment of student achievement. We are 
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pleased that principals and teachers whom the auditor interviewed believe that 
consistency in assessment is improving in their individual schools. 


The Ministry will work with boards to develop ways to monitor and report on 


consistency in student assessment. We will research and develop ways to 
ensure greater consistency in final evaluations of student achievement and 
research appropriate outcome measures. We will continue to consult with © 
school board leaders on how to enhance accountability measures related to 
curriculum implementation. 








STRENGTHENING IMPROVEMENT PLANNING 
AND RESEARCH 


At present, EQAO test results represent the only comprehensive data available on the 

Pp P Vi Pp 

performance of students across the province. The table below presents a summary of the 
results since the tests were introduced. 


EQAO Results—Students Who Achieved at Levels 3 and 4 as a % of Enrolment 
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In response to EQAO results, schools, boards, and the Ministry have taken steps to improve 
student performance on EQAO tests. Some steps provided a one-time increase in the level 
of student performance. For example, we were told by educators we interviewed that 
designing classroom problems in a format that is substantially the same as that used on 
EQAO tests, so that students are familiar with the format, improves student performance 
on the tests. The impact of such initiatives on student performance appears to be 
diminishing since, as can be seen in the table, there has been little improvement in the 
proportion of students performing at or above the provincial standard (level 3) after the 
third year of testing. 
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Other steps taken that are intended to contribute to a long-term improvement in 
performance include: 


Boards we visited provided some additional resources to their lowest-performing schools, 
such as more time from consultants or other assistance, to facilitate the implementation 
of instructional practices that were better suited to the students concerned. For example, 
we visited an elementary school that had been among the lowest-performing schools in 
the province on the EQAO tests. The school changed its approach to language 
instruction and purchased a program, including textbooks, that supported the new 
approach, and subsequently experienced a significant improvement, as anticipated, on 
the EQAO reading and writing tests. In addition, the school achieved much better 
results on the math test, which the principal and teachers believed resulted from their 
students being better able to understand the questions due to their improved language 
skills. In addition to school-based initiatives, EQAO results also provided the impetus for 
some boards to implement remedial programs, such as summer literacy camps, and 
tutoring programs whereby high-performing students in senior secondary grades assist 
students in junior secondary grades who are having problems. 


The Ministry used EQAO results to identify schools that were eligible for assistance 
under its Support for Schools That Need Extra Help Program. This three-year, 

$5 million pilot program was announced in October 2001. Schools where less than 
one-third of students have achieved at levels 3 and 4 on the EQAO grade 3 test for 
three consecutive years are eligible for assistance. The Ministry has selected 29 of the 58 
eligible schools identified so far and plans to increase the number to 40-45. Under the 
program, specialist teams composed of former principals and university experts assist 
selected schools in diagnosing the reasons for their poor performance and in developing 
strategies for corrective action. The schools base their improvement plans on this 
information, submit them to the Ministry, and each may receive financial assistance over 
three years to assist them to implement their plans. 


Further gains in student performance will have to be achieved through school and board 


improvement planning processes that result in the causes of poor student performance 


being identified and appropriate corrective action being implemented, such as changes in 


instructional strategies and techniques. At the time of our audit, however, none of the 
boards we visited had established effective improvement planning processes due to the 
absence of certain key requirements, such as: 


sufficient, accurate, and comparable student performance and contextual data useful 
for interpreting achievement results as well as information systems that are capable of 
supporting the collection and analysis of such data. The Ministry has not established 
standards regarding the minimum capabilities and content of student information 
systems. As a result, we found, for example, that the boards we visited did not record 
data on their student information systems such as participation in remedial programs or 
the failure to complete homework (identified in one study as “the single biggest 
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detriment to student achievement’). As noted earlier in this report, neither the Ministry 
nor the boards have procedures in place to provide assurance that student performance 
data is comparable; 


* appropriate training to ensure that school and board personnel have the knowledge and 
skills necessary to use the systems and perform the analyses needed to identify and 
prioritize problems, develop viable plans with challenging but realistic objectives, 
establish appropriate interim milestones to measure progress in achieving these 
objectives, and measure and report on the extent to which the plans’ objectives have 
been achieved; and 


* procedures to assess the quality of the improvement planning processes in place at 
schools and at school boards, and to ensure that corrective action is taken where 
deficiencies in the processes are identified. At the time of our audit, the boards we 
visited had not established formal review processes for their schools’ improvement plans, 
although one board was planning to do so with respect to the plans for the 2003/04 
school year. 


Every school board is required to submit a board improvement plan to the EQAO that 
meets specific content requirements established by the agency. The EQAO reviews and 
provides comments on each board’s improvement plan in relation to its published criteria. 
However, these reviews do not include such matters as determining whether problem 
identification is consistent with the EQAO results for the board overall and for its schools, or 


whether appropriate progress reporting procedures are in place. 


Research to Support Decision-making 


In addition to its impact on improvement planning processes, the lack of sufficient, 
comparable student performance data and suitable computer support systems to capture 
and analyze contextual data also limits the ability of the Ministry and school boards to 
conduct the research necessary to address critical issues in curriculum delivery and to 
provide the basis for informed decision-making. While the EQAO has established an 
Education Quality Indicator Program to collect some contextual data that is intended to 
help readers interpret student achievement results it publishes, the data is useful primarily in 
identifying areas of focus for further study and research. However, the Ministry had not, 
either directly or through the EQAO, routinely sponsored research or co-ordinated 
educational research carried out by school boards and the province’s universities. Given that 
several school boards have established their own research and evaluation functions, some co- 
ordination of their efforts would increase the cost effectiveness of research and benefit all 


school boards. 
There are many significant issues that research could help address. For example: 


* Even though the correlation between lower socioeconomic status and poor performance 
is well documented, it is not inevitable. We found a number of reports of schools in 
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Ontario and other jurisdictions where students coming primarily from economically 
disadvantaged families performed well. Research that identified the factors that actually 
cause this relationship would assist educators in determining the strategies that they 
should employ to improve the performance of these students. 


¢ Some educators have suggested that larger classes for high-performing students and 
smaller classes for low-performing students who require greater assistance would increase 
the number of students who meet provincial expectations. 


¢ While some educators believe that using subject specialists rather than generalists to 
teach grade 7 and 8 students improves student achievement, the extent to which this is 
true and whether any observed improvement in student achievement carries through to 
students’ grade 9 report card marks and their marks on the EQAO grade 9 math test 
and the OSSLT have not been studied. 


¢ The College of Teachers and the Task Force on Effective Schools—as well as research 
that we reviewed—concluded that teacher quality is the most important factor 
influencing student achievement. While the Ministry and school boards are at present 
implementing a new teacher-performance evaluation process, the Ministry has not 
implemented methods of tracking the impact of teaching quality on student 
achievement, as is done in some other jurisdictions (for example, the Tennessee Value- 
Added Assessment System). A recent Canadian study, “Teacher Quality in Canada,” 
stated that “Canada has not yet utilized multi-year comprehensive student achievement 
data in order to measure teacher effects on learning or value-added achievement gains. 
Even where such data are available, we do not yet have a culture within the public 
education system that fully recognizes its potential value.” 


Clearly, there are many issues that school boards and the Ministry need to examine if 
significant improvements in student performance are to be achieved. A co-ordinated effort 
will help ensure that research is conducted cost effectively. In this regard, in July 2003, the 
EQAO initiated an external research program in which proposals for reports and analyses 
on topics relevant to the work of the EQAO were solicited. The research program may also 
include critical literature reviews that have implications for policy on assessment and 


educational improvement. 










Recommendation 


To help ensure that decisions regarding curriculum delivery are based upon 
sufficient and reliable information, and to enhance the effectiveness of the 
improvement planning process, the Ministry should: 





e establish standards regarding the capability of student information 
systems that school boards use and the information that is recorded on 
them; 

¢ co-ordinate and support training for school and board personnel in 

implementing effective improvement planning processes; 
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e implement, either through the Education Quality and Accountability Office 
or otherwise, a review function for school board and school improvement 
planning processes that includes on-site examination; and 


e co-ordinate and support research on key curriculum delivery issues. 


Ministry Response 


Research, data management, and information systems support effective 
decision-making and improvement-planning processes. 


The Ministry has trained board teams on how to use assessment data to inform 
improvement planning and sponsored three regional symposia on “Data 
Driven Decision-Making for School Improvement.” Approximately 2,000 school 
board administrators were trained to use quality data to support improved 
decision-making. 


The Ministry is developing standards to guide boards in their data 
management. The Ministry and the Education Quality and Accountability Office 
will also work with the Council of Ontario Directors of Education to determine 
cost-effective ways to improve school improvement planning processes. We 
will continue working with the education sector to ensure that decisions 
regarding curriculum delivery are based upon sufficient and reliable 
information. In addition, the Ministry will commit to continuing support for 
research, dependent on the availability of appropriate resources. 





EVALUATING THE ANNUAL EDUCATION PLAN/ 
TEACHER ADVISER PROGRAM 


In order to help meet the objective of creating “effective links to work and higher 
education,” the Ministry included a for-credit career education course in the curriculum 
and required school boards to offer co-operative education/work experience opportunities, 


and to provide the Teacher Adviser Program (TAP) and the Annual Education Plan (AEP) 
for their students. 


In September 1999, the Ministry began requiring that principals establish the TAP for 
students in grades 7 to 11, with the option of providing the program for students in grade 
12. In July 2001, the Ministry revised the policy to encourage but not require the 
assignment of a teacher adviser for students in grades 11 and 12. The AEP was also 
introduced by the Ministry in 1999 for students in grades 7 to 12 and was designed to help 
students take responsibility for their education, make informed decisions, and plan for their 
futures with the help of parents, teacher advisers, and guidance counsellors. 


At the elementary level, teachers and principals felt that these initiatives were beneficial 
because they encouraged students to think about the fact that what they do in school has an 
impact on their future. Secondary teachers and principals generally agreed with this benefit, 
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but most felt that the AEP and TAP programs were not effective as implemented for the 


following reasons: 


¢ Classroom teachers did not feel that they had the knowledge required to provide 
students with advice regarding their career choices—this is the role of guidance 
teachers. They also stated that, given the timetabling realities of the typical secondary 
school, it is impossible to obtain the continuity that is required for teachers to know 
enough about their students to function effectively as advisers. 


¢ Principals and teachers stated that students have no immediate incentive to take the 
process seriously. They told us that students are motivated by marks. Since there are no 
marks associated with the AEP and TAP programs, students will not put much effort 


into them. 


The majority of educators at the secondary level were of the opinion that it would be better 
to reallocate the resources attached to these initiatives to other areas—for example, more 
guidance teachers—as opposed to trying to fix the two programs. 


In addition to the concerns raised by educators, we noted that: 


¢ The Ministry has not established measurable objectives for the program, or developed 
any procedures to monitor its effectiveness. 


e At the secondary schools that we visited: 


- AEPs were not critically assessed by teacher advisers. Thus, students are allowed to 
state that they plan to do better in math without an adequate explanation as to how 
this is to be accomplished other than the generic “work harder” or “study more.” 


- Inthe AEPs that we reviewed, students were not asked to note in their AEPs any of 
the post-graduation destinations that they were considering. Thus there was no link 
between the AEP and career objectives and no basis for advising, for example, a “C” 
student in math that some of his or her career destinations typically require a B+. 


- There is no formal process to monitor progress on achieving the objectives noted in 


the AEP. For example, if improving his or her mark in math was a student's 
objective, we would expect to see milestones established, such as a trend of 
improving marks on assignments and midterm tests and, if the milestones were not 
met, evidence of a discussion of the corrective action (that is, remedial work of some 
sort) that was taken. 








Recommendation 


In order to help ensure that appropriate benefits are realized from the Annual 
Education Plan/Teacher Adviser Program, the Ministry should, in conjunction 
with school boards and principals, formally assess the success of the program 
in meeting the needs of the students. If the assessment is positive, measurable 
objectives for the program should be established. 
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Ministry Response 


The Annual Education Plan and Teacher Adviser Program is considered an 
important tool to help students achieve their educational goals. 


Ministry policy already requires school principals to conduct a survey every 
three years with students, parents, teachers, and community members to 
determine the effectiveness of the school’s guidance and career education 
program, including the Annual Education Plan and Teacher Adviser Program. 


The Ministry will undertake a review of the implementation of the Annual 
Education Plan and Teacher Adviser Program in Ontario schools. This review 
will involve key education stakeholders, including principals. 
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MINISTRY OF ENTERPRISE, OPPORTUNITY 
AND INNOVATION 


3.06—Business and 
Economic Development 
Activities 





BACKGROUND 


The Ministry of Enterprise, Opportunity and Innovation was created on April 15, 2002 


with the amalgamation of the science and technology activities of the former Ministry of 


Energy, Science and Technology with the business and economic development activities of 
the former Ministry of Economic Development and Trade. The Ministry’s mandate is to 
foster an Ontario with competitive businesses and a prosperous economy by promoting 


innovation, economic growth, and job creation. 


The Ministry has four divisions employing 300 staff who deliver its business and economic 


development activities: 





The Competitiveness and Business Development Division supports business 
development, manages the government's relationship with various industry sectors, and 
delivers the Strategic Skills Investment program, which provides financial support to 
training institutions to develop job skills for Ontario’s labour market. 


The Investment Division attracts investment to the province, marketing it as a premier 
investment location using advertising and promotion and by generating business leads 
and providing investment services. 


The Trade Development Division is responsible for increasing Ontario’s global exports 
by working with Ontario-based firms (mainly small- and medium-sized enterprises) to 
expand their exports worldwide. A government agency, Ontario Exports Inc., carries 
out most of the responsibilities of this Division. 


The Corporate and Field Services Division offers financial and management advice to 
businesses and entrepreneurs and operates a network of field offices and small-business 
enterprise centres to promote growth, export activity, and job creation. 
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For the 2002/03 fiscal year, the Ministry’s funding for its business and economic 
development activities was $78.9 million, which was allocated to the four divisions as 
outlined in the following chart. 


Business and Economic Development Expenditures by 
Ministry Division, 2002/03 


($ million) 
Corporate and Field 
Services Division 
$10.5 Competitiveness and 
Business 
Development 
Divisi 
Trade Development eas 
Division : 
$14.3 





Investment Division 
$24.2 


Source of data: Ministry of Enterprise, Opportunity and Innovation 


AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of the business and economic development activities of the 
Ministry of Enterprise, Opportunity and Innovation were to assess whether it had adequate 
procedures in place to: 


* measure and report on the activities’ effectiveness in promoting competitive businesses 
and a prosperous economy; and 


e ensure that resources are managed with due regard for economy. 
ag g y. 


The criteria used to conclude on our audit objectives were discussed with and agreed to by 
ministry management and related to systems, policies, and procedures that the Ministry 


should have in place. 


Our audit was performed in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants, and accordingly included such tests and other procedures as we 
considered necessary in the circumstances. 


The scope of our audit fieldwork, which was substantially completed by March 2003, 


included discussions with relevant staff, as well as a review and analysis of documentation at 
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the head offices of the Ministry and of Ontario Exports Inc. We also visited several training 
institutions that were recipients of Strategic Skills Investment grants to review their 
implementation of training programs. 


Our audit also included a review of the activities of the Ministry’s Internal Audit Services 
Branch. The Branch had recently carried out audits of the Strategic Skills Investment 
program, employee travel expenses, and consulting services. We reviewed these audit reports 
and adjusted the scope of our work where possible. 


OVERALL AUDIT CONCLUSIONS 


We concluded that the Ministry did not have the necessary strategic planning processes and 
information systems to support training institutions in addressing skills shortages and to assist 
Ontario businesses in expanding their export potential. Furthermore, we noted a number of 
instances where procedures to ensure due regard for economy needed to be improved. For 
example, we noted the following: 


¢ The Ministry had not evaluated the Strategic Skills Investment (SSI) program to 
determine whether the program was successful in addressing the current and 
anticipated skills needed to ensure business competitiveness in Ontario and to 
ensure that students obtained employment in their respective areas of training. The 
Ministry had created a database for the intended purpose of measuring the 
program’s success, but we found that the system could not generate reliable 
information. Furthermore, approximately 75% of ministry SSI funding that we 
reviewed was used for construction and equipment costs rather than for the direct 
provision of skills-based training. 


¢ The Ministry had not developed methods to measure the extent to which it had 
achieved its objective of promoting innovation, economic growth, and job creation. 
Instead, the Ministry tended to assess its performance by measuring activities. For 
example, one of the ways the performance of the Trade Development Division was 
evaluated was by monitoring the number of clients assisted rather than by assessing 
whether any increases in export trade had actually occurred. 


¢ Akey to achieving success with the Investment Division’s international marketing 
campaign is ensuring that adequate analysis of advertising media, target markets, and 
target sectors is performed before the marketing plan is developed. We found that the 
advertising and marketing campaign was well planned and that appropriate research 
was carried out to support the development of a focused marketing plan. 
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DETAILED AUDIT OBSERVATIONS 


COMPETITIVENESS AND BUSINESS 
DEVELOPMENT 


The Competitiveness and Business Development Division supports economic growth and 
job creation by fostering innovation and business development, negotiating trade 
agreements, promoting the removal of barriers to business, and providing businesses and 
entrepreneurs with information and advice relating to economic development and trade 
policy. The Division also manages the government’s interactions with a variety of business 
sectors, including the aerospace, automotive, manufacturing, and retail sectors. 


We reviewed the transfer payments made by the Division to the various sectors it supports 
and determined that payments were properly supported and appropriately classified in the 
Ministry’s financial records. 


Strategic Skills Investment Program 


In addition to the activities described above, the Division also manages the Strategic Skills 
Investment (SSI) program, which supports training programs and courses that accelerate 
skills development and are offered by training providers in collaboration with businesses. 
The training providers involved include community colleges and private training 
institutions. Businesses provide financial and in-kind contributions to training providers in 
support of specific courses. The main focus of the SSI program is on generating the skills 
needed to build business competitiveness and enhance the responsiveness of Ontario’s 
training institutions to produce graduates with skills needed by Ontario businesses. 


Since 1998, when the SSI program was established, the Ministry has committed a total of 
$130 million to it. Since that time, the Ministry has spent $82 million on 48 training 
programs. 


We examined SSI program files, reviewed the evaluation process, and visited four training 
providers. We determined that the Ministry needed to better identify and respond to skills 
shortages and improve its procedures and processes for evaluating program funding 
proposals submitted by training providers. 


IDENTIFYING AND RESPONDING TO SKILLS SHORTAGES 


The primary objective of the SSI program is to address current and anticipated skills needed 
to ensure business competitiveness in Ontario. In August 1999, the Ministry informed the 
Management Board of Cabinet that it would measure the SSI program’s success by the 
number of skilled employees available for participating industry sectors. This measure would 
be determined by the number of students enrolled in SSI training, the number successfully 
completing the training, and the number obtaining employment in the related line of work. 
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To accomplish this, the Ministry created a database and required training institutions to 
annually submit data certifying the number of students enrolled in SSI-funded projects and 
the graduate placement rate. We reviewed this database for the three years ending 

March 31, 2002 and found that we could not reconcile the information generated by the 
database to the information provided by the training institutions. The Ministry attributed 
these concerns to a number of factors, including a poorly designed database, improper data 
definition, and incomplete submissions. The Ministry had not followed up on these 
concerns to ensure that the database was accurate, complete, and useful for the intended 
purpose of measuring the program’s success in addressing strategic skills shortages. 


In addition, we reviewed 29 of the 48 training projects funded by the Ministry up to 
March 2003. The Ministry contributed $48 million towards the total costs of these projects. 
We noted that 75% of the SSI funding reviewed was used for construction and equipment 
costs, while only 25% of the funding went to actually providing skills-based training. In 
addition, we noted six projects where a total of $11.8 million in ministry funding was used 
for construction and equipment and no funds were allocated to direct training. While some 
infrastructure costs are necessary, we question whether the Ministry is adequately 
responding to skills shortages when such a high percentage of program funds is spent on the 


training institutions’ infrastructure instead of on the provision of training. 





Recommendation 


To ensure that the Strategic Skills Investment program adequately supports 
the development of the strategic skills necessary to enhance business 
competitiveness, the Ministry should: 


e resolve its database concerns to determine whether the program is 
adequately addressing skills shortages and whether graduates are 
employed in the areas for which they were trained; and 

e review the reasonableness of the percentage of ministry funding that is 
used for construction and equipment costs instead of for direct training 
costs. 





Ministry Response 


The Ministry monitors individual projects with regard to student enrolment and 
completion numbers as part of ensuring that each project is meeting its 
milestones. The Ministry agrees, however, that database development concerns 
have significantly hampered its capacity to provide aggregate information 
about projects collectively. Strategic Skills Investment (SSI) staff are 
continuing to work with the Ministry’s information technology specialists to 
develop a database to readily summarize data and simplify monitoring of 
overall results. 


The SSI program only supports one-time start-up costs, including those for 
new space, new equipment, and new curriculum. In light of this 











Business and Economic Development Activities 149 


| VFM Section 3.06 | 





| VEM Section 3.06 









recommendation, the Ministry agrees to review the reasonableness of the 
percentage of SSI funding that is used for construction and equipment costs 
instead of for direct training for each project. 






EVALUATING AND PROCESSING PROPOSALS 


In 1999, the government anticipated that the SSI program’s contribution to strategic skills 
training would amount to approximately 25% of total funding. The remaining funds would 
be generated from private-sector partners. Although the Ministry stipulated that private- 
sector contributions should be significant, it did not establish clear funding guidelines or 
impose limits on its funding percentage for individual projects. We noted that while the 
Ministry's funding averaged about 25%, its contributions to individual SSI projects ranged 
from a low of 3% to as much as 62% of the total cost of training projects funded. In many 
cases, the Ministry did not meet one of the program’s objectives—that of generating 
significant contributions from industry partners. 


For contributions that were made by industry partners, the Ministry recognized both cash 
and cash equivalent or in-kind funding, such as donations of goods, services, labour, or 
equipment. In attributing a value to the contributions, consumable goods and services are to 
be valued at cost and are not to include any markup, and equipment is to be valued at the 
lesser of fair market value and the depreciated book value. We noted that the Ministry 
relied on applicants to value in-kind contributions. No documentation was included in the 
Ministry’s files regarding an assessment of the reasonableness of valuation. In our visits to 
recipient training providers, we noted that for equipment valuations, there was little 
independent documentation supporting the value amounts. These amounts can be 
significant: at three recipients we visited, they totalled $5.7 million. In a recent audit of the 
SSI program, the Ministry’s Internal Audit Services Branch also expressed concerns about 
the valuation of in-kind contributions without an independent appraisal. 


Another area we reviewed was the timeliness of processing applications. Because the 
demand for specific skills can quickly change, it is important that the Ministry fund projects 
on a timely basis. For 2002, the Ministry planned to issue two calls for proposals from 
training providers. Normally, the Ministry’s review and approval process takes from two to 
three months. However, we noted that applications submitted by training providers before 
the first proposal deadline of June 20, 2002 were not approved until February 3, 2003— 
more than seven months after the proposal closing date. Because of this delay, the planned 
call for proposals in the fall of 2002 never took place. Consequently, the Ministry did not 
meet its expected timetable of two calls for proposals in the 2002/03 fiscal year and was 
therefore unable to fund skills training to the extent originally anticipated. 
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To ensure that the Strategic Skills Investment program increases the 
responsiveness of training institutions to meeting business needs, the Ministry 
should: 


e ensure that in-kind contributions from private-sector partners are properly 
valued and, where amounts are significant, request independent valuations; 

e review its share of funding for training programs in relation to the share 
provided by industry partners and consider developing funding-level 
guidelines; and 

e ensure that training program proposals are evaluated, processed, and 
approved on a timely basis. 








Ministry Response 


The Ministry agrees that consistent documentation of the value of in-kind 
contributions is important and is evaluating the best practices of Strategic 
Skills Investment (SSI) projects with the intent to incorporate those best 
practices in SSI program guidelines for use by all future proponents. In 
particular, the Ministry will implement a process to independently verify 
contributions valued at more than $100,000. 





The current SSI guidelines call for industry contributions to be substantial in 
light of the project. While the current average for all projects puts SSI's 
contribution at 25%, with 75% coming from project partners, the Ministry will 
commit to a guideline of no more than 50% SSI contribution to any individual 
project. 





The Ministry agrees that proposals should be processed on a timely basis. The 
Ministry will set a target date for completing the processing of each call for 
proposals. 





EXPORT TRADE AND INVESTMENT 


Part of the Ministry’s mandate is to expand Ontario’s export markets and to market Ontario 
as a premier investment location, attracting investors to locate and expand their operations 
in the province. This is accomplished mainly through the Ministry's Trade Development 
and Investment divisions. 


Trade Development 


A government agency, Ontario Exports Inc., carries out most of the responsibilities of the 
Trade Development Division. The agency works with Ontario companies to develop new 
export opportunities and assists foreign buyers to find Ontario suppliers for their purchasing 
requirements. In pursuing these aims, the agency leads and participates in a series of 
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initiattves—including international trade missions, exhibitions, and seminars—and provides 
services in export market intelligence, consulting, and commercial advocacy to expand the 
international trade potential of Ontario firms. The agency’s clients are mainly small- and 
medium-sized Ontario enterprises. 


We reviewed the agency's direct operating expenditures for the 2001/02 fiscal year and 
noted that approximately 30% of the agency’s funding for export activities was directed to 
initiatives in the United States. The remaining 70% of the agency’s funds was directed to 
other international markets. The Board of Directors had instructed the agency to diversify 
Ontario away from dependence on the U.S. market, given that more than 90% of Ontario’s 
exports are to the United States. 


Long-term forecasts of Ontario’s trading export potential indicate that the U.S. will remain 
the province’s primary market. For instance, in October 2002, the Conference Board of 
Canada issued a report entitled Canada 2010: Challenges and Choices at Home and Abroad. 
The report indicates that our business linkages will be driven by our relationships within 
North America. It further states that “ensuring access to the U.S. market is, arguably, the 
single most important policy consideration. ..in the coming decade.” In March 2002, 
Export Development Canada had also reported that: the United States will remain a 
growing and dominant destination for Canadian exports; the expansion of the European 
Union will make it even harder for Canadian businesses to maintain or grow their market 
share in Europe; exports to Japan are not expected to grow in the near future; and 
Canadian exports to China have been stagnant since 1995. 


We also reviewed the agency’s trade development activities, which include trade missions 
and trade shows, and noted that the agency did not evaluate the success of these activities in 
generating export business for Ontario companies. As a result, budget allocations to the 
various target markets were based more on what was allocated in previous years than on an 
analysis of potential costs and benefits. In addition, a November 2001 review of the 
corporation reported survey results whereby clients expressed postive feedback on the 
corporation but questioned the effectiveness of some of the corporation’s trade development 
activities. Corporate staff were also surveyed and expressed concerns that some activities 
consumed a disproportionate amount of the corporation’s resources and that these activities 
might be more suited to larger firms rather than the small- and medium-sized businesses 
that are the corporation’s strategic focus. To better support Ontario firms in expanding their 
export potential, the Ministry needs to conduct a more formal evaluation of export 
opportunities; and to help maximize its marketing efforts, it needs to assess the costs and 
benefits of previous initiatives. 


Investment 


The Investment Division markets Ontario as an investment location’to domestic and 
international business decision-makers. It does so by advertising, generating business 
investment leads, and providing investment services to potential investors. These services 
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include providing detailed economic information about Ontario’s potential as an investment 
location and assisting prospective investors in preparing business cases to support investment 
decisions. The agency’s major clients are multinational corporations located in Ontario that 
may consider expanding and foreign-based firms considering new investments. 


A key to achieving success with the Division’s international marketing campaign is ensuring 
that adequate analysis of advertising media, target markets, and target sectors is performed 
before the marketing plan is developed. We determined that the advertising and marketing 
campaign was well planned and that appropriate research was carried out to support the 
development of a focused marketing plan. In addition, the leads generated from advertising 
and by in-market business development consultants (consultants hired by the Ministry to act 
as representatives in foreign target markets) were tracked and reviewed by management 
before these were transferred to the Division’s International Investor Services section, which 
is responsible for attracting and retaining domestic and foreign investment. 


The International Investor Services section maintains a list of potential business investment 
leads and related information, including company name, country of origin, nature of 
business operations, and the relevant business sector. However, it does not track the source 
of the lead—that is, whether it comes from in-market business development consultants, 
advertising, the federal government, municipalities, or the companies themselves. Such 
information would be useful to assess the effectiveness of the various investment initiatives 
undertaken by the Division to generate leads, as well as to improve its strategic planning and 
allocate resources based on the source of such leads. 



















Recommendation 


To more effectively support the Ministry’s initiatives for increasing Ontario 
exports and attracting investment to the province, the Ministry should: 


¢ perform a formal analysis of export opportunities, assess the success of 
the previous initiatives of Ontario Exports Inc., and conduct a 
comprehensive cost/benefit analysis of this agency’s trade development 
initiatives; and 

e track the source of business investment leads generated in the Investment 
Division to help improve its strategic planning and resource allocation. 


Ministry Response 


The Ministry completed a program review of Ontario Exports Inc. (OEl) in 2001. 
As a result of the review, the corporation undertook a planning process to 
identify and evaluate the economic business environment, key market 
opportunities, client demand, other export services available, and resource 
implications. A three-year strategic plan for the period 2003/04 to 2005/06 was 
approved by the OEI Board. The plan identifies market and sector priorities and 
key activities for each market and sector. The plan also balances the 
importance of the U.S. markets with the need to make a long-term commitment 
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to develop other important markets. The resources have been allocated on the 
basis of these priorities. 







As a further follow-up to the program review, OEl is currently undertaking 
research to develop an economic impact index to measure the economic 
contribution of its export advisory services. The research will focus on the 
impact of exports by small- and medium-sized enterprises on the provincial 
Gross Domestic Product, tax revenue, and employment. The final report, which 
is due in August 2003, will provide OEI with a tool for measuring its programs’ 
economic impact. 


The Ministry will add the source of the investment lead to its investment 
tracking program. 





TRADE DEVELOPMENT AND FIELD SERVICES 
GRANTS 


Grants are given to organizations for a defined purpose. Ministry managers are therefore 
required to obtain and review information on the performance of recipients to ensure that 
grants are being spent for the intended purpose and to determine whether the expected 
results are ultimately being achieved. Such monitoring may involve informal contact with 
recipients, review of records, site visits, or audits. Where recipients have failed to comply 
with the terms of their agreement with the province, managers are to initiate corrective 
action to ensure that funds are used as specified in the agreement. We reviewed the Trade 
Development Division's grants to the Canada Science and Technology Centre in Jiangsu, 
China. We also reviewed two entrepreneurship grant programs—the Summer Company 
program and the My Company program—that are delivered by the Corporate and Field 
Services Division. As outlined below, we found that the Ministry needed to improve its 
monitoring efforts to ensure that the objectives of grant programs are met. 


In 1985, the Ministry entered into an agreement with the Canada Science and Technology 
Centre in Jiangsu, China, to help promote and exchange ideas with respect to economic 
development and trade. The agreement was that operating costs would be shared equally 
between the province and Jiangsu. As of March 31, 2003, ministry payments to the Centre 
since the inception of the agreement totalled $837,000—approximately $65,000 annually. 
Aside from a ministry review of invoices for its share of the operating costs, we found no 
evidence that the Ministry adequately monitored the recipient to ensure that funds were 
used as agreed. In May 1999, the Ministry’s Internal Audit Services Branch issued a report 
of an audit carried out on the Centre. The report revealed that: most of the building space 
was vacant and not being used; a database of Ontario and Jiangsu companies had not been 
established as set out in the agreement; and an Ontario showroom set up to provide 
information on Ontario companies no longer existed. In 2001/02, the Ministry suspended 
payments to the Centre. A ministry review in March 2002 concluded that Ontario had not 
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benefited from the 17-year relationship and was not getting value for the resources invested. 
At the end of our audit, in March 2003, although payments to the Centre had been 
suspended, the relationship with the Centre was still in place and the Ministry was still 
budgeting funding for the Centre. 


The Summer Company program provides $3,000 grants to students to gain experience in 
entrepreneurship by running a small business over the summer. Grants under this program 
for the 2002/03 fiscal year totalled $654,000. The Ministry monitors the progress of 
participants over the summer and requires them to complete a final report on their 
experience. However, there is no formal assessment of how successful the program has been 
in meeting the Ministry’s objectives of providing the tools and experiences necessary to 
enhance the success of small business start-ups by young entrepreneurs and of promoting 
entrepreneurship as a viable career option. 


The My Company program provides guarantees of up to 85% for loans of $15,000 or less 
to young entrepreneurs starting their own business. This is a partnership arrangement with 
one of the chartered banks. For the 2002/03 fiscal year, the Ministry guaranteed loans 
issued by the bank totalling over $400,000. The program guidelines indicate that the loan 
may not be used to finance existing debt, purchase an existing business and assets, operate 
an extension of an existing business, or purchase a major item, such as a vehicle or computer. 


The bank is responsible for selecting, approving, and awarding loans, as well as monitoring 
the applicants. While the Ministry receives quarterly reports from the bank listing the 
outstanding loan amounts and the status of each loan, we noted that it did not verify nor 
receive assurance from the bank that the loans were used for the intended purpose. 






Recommendation 


















To help meet program objectives for grant programs, the Ministry should 
ensure that proper systems for monitoring recipients are in place and that the 
success of each program is formally evaluated and taken into consideration in 
future funding decisions. 





Ministry Response 


The Ministry agrees that proper monitoring systems should be in place to 
support the evaluation of grant programs. 


Ontario Exports Inc. has now completed an assessment of the $65,000 grant to 
the Canada Science and Technology Centre in Jiangsu, China, and determined 
that there are more effective activities to promote exports to China. The 
Ministry will continue to work with the Centre to promote Ontario-Jiangsu 
exports but will no longer provide direct support to the Centre. The grant line 
will be eliminated from the Ministry’s 2004/05 expenditure estimates. 


Summer Company and My Company are initiatives under a new program, the 
Youth Entrepreneurship Program, which was established in 2001 to promote 
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entrepreneurship among Ontario’s youth. The Ministry is implementing a 
process for tracking the program’s results, including monitoring the recipients. 
The program was awarded a Gold Public Sector Quality Award and is 
continually working to improve its effectiveness. 













MEASUREMENT OF AND REPORTING ON 
PROGRAM EFFECTIVENESS 


‘To demonstrate that its programs and policies for business and economic development are 
effective in accomplishing its mandate of fostering an Ontario with competitive businesses 
and a prosperous economy, the Ministry needs mechanisms to track the results of its 
initiatives and to take corrective action where objectives are not met. 


Prior to 1995, the Ministry operated many direct business-assistance programs that 
provided grants, financing, loans, and loan guarantees—primarily through the Ontario 
Development Corporation. Since then, the Ministry’s operational focus has been less on 
direct financial assistance and more on functions like advising companies, helping them 
access the resources needed to compete, marketing the province as an investment location, 
and helping Ontario exporters find new customers in world markets. While it can be more 
difficult to assess the impact of such activities, the Ministry is nevertheless required to 
measure and report annually through the business planning and allocation process on its 
effectiveness in achieving its mandate. 


Performance measures should not report on ministry activities; they should show the likely 
economic results of those activities. Many of the Ministry’s current measures are based on or 
defined as activities—for instance, the number of projects approved and the number of 
clients and companies assisted. Such measures describe activities rather than outcomes or 
economic results. For a performance measure to be relevant, it must show the direct 
relationship between the activity and the resulting outcome. For example, one of the 
Ministry's performance measures for its trade program was the number of clients assisted to 
maintain their export markets and expand internationally. While this may measure the level 
of ministry activity in this area, it does not measure success—that is, whether export trade 
actually increased as a direct result of ministry assistance. As another example, the key 
performance measure for the Strategic Skills Investment program was the level of employer 
satisfaction with the program. While the program received a 90% satisfaction rating, this 
does not measure the program’s objective, which is to help develop the skills necessary for 
business competitiveness. This objective directly relates to the Ministry’s overall mandate of 
fostering competitive businesses. The Ministry did not report on whether this objective was 
accomplished. 


The Ministry noted that measuring program results for economic development and trade 
can be difficult and may not always be within the direct control of its programs. Other 
parties can influence results, including the federal government and multinational 
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companies, as can other factors, such as domestic and world economic conditions, trade 
barriers, and international events. Nevertheless, Ministry staff acknowledged that even 
though specialized assessments may be required, more appropriate performance measures 
were needed to evaluate and report on the effectiveness of its business and economic 
development activities. 


Recommendation 


To measure and report on the effectiveness of its business and economic 
development activities, the Ministry should develop performance measures 
that demonstrate how program initiatives contribute to the fostering of 
competitive businesses and a prosperous economy. 


Ministry Response 


The Ministry tracks the performance of its business and economic 
development activities. For example, extensive tracking is done on the 
effectiveness of the investment marketing and servicing activities. 


The Ministry is working to improve the measurement of the impact of these 
activities. For example, Ontario Exports Inc. is currently undertaking research 
to develop an economic impact index to measure the economic contribution of 
its export advisory services. The Ministry also periodically surveys other 
jurisdictions for their performance measures for similar programs. The Ministry 
also works with central agencies on the development of performance 
measures for new or revised programs. 


The Ministry is committed to measuring the contribution of its programs and 
will continue to refine and improve performance measures for its business and 
economic development activities. 


TRAVEL EXPENDITURES 


Travel expenditures that are incurred by employees on ministry business and that are 
eligible for reimbursement relate to such areas as transportation, accommodation, car 
rentals, meals, and hospitality expenses. Management Board of Cabinet directives outline 
the general rules all ministries and government employees must follow to minimize the cost 
of travel expenses. During the two years ending March 31, 2003, the Ministry incurred 
over $3.5 million for travel expenses. We reviewed the travel claims process for these two 
years and noted numerous examples where employees disregarded Management Board of 
Cabinet directives and managers did not adequately scrutinize travel expenses. 


When incurring travel expenses for such items as hotels and personal meals, employees are 
required to follow Management Board of Cabinet directives. However, for meal allowances, 
Management Board of Cabinet directives are silent on acceptable rates outside Ontario. 
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Nevertheless, the directives require that persons approving travel claims ensure that meal 
costs are reasonable for the locations in which they are incurred. As a proxy for reasonable 
meal rates in foreign jurisdictions, we compared the federal government’s meal allowances 
with a sample of ministry claims and noted more than 100 cases where employees claimed 
personal meal costs in excess of these rates, and the excess averaged 50% more than the 
federal rates. 


Before approving travel expense claims, managers are required to scrutinize travel expense 
claims to ensure that all expenses claimed are necessary, reasonable, and supported by 
appropriate receipts. However, we found that managers did not adequately scrutinize travel 
expense claims. We noted several instances where employees had been reimbursed for 
expenses without receipts or with receipts that were inappropriate, such as photocopied 
receipts and credit-card statements instead of original receipts. In such cases, there is a risk 
that ineligible expenses could be reimbursed or duplicate payments could be made. In fact, 
we noted two duplicate payments totalling approximately $1,200 where ministry staff were 
reimbursed for expenses using photocopied receipts. We informed the Ministry of these two 
instances and the duplicate payments were subsequently recovered. In addition, we found 
instances where managers approved the reimbursement of costs that were questionable 
business expenses, such as the renewal fee for a personal driver’s licence and payment for a 
spouse’s admission to an air show. 


Employees extending hospitality to business clients and contacts are required to submit 
receipts detailing the amounts paid for food, beverages, and other items, as well as to 
indicate the persons in attendance. This is to ensure that only business clients and persons 
who are not employed by the government are extended hospitality. We noted over 200 
cases, totalling over $48,000, where hospitality costs were claimed without detailed receipts. 
Supporting documentation consisted primarily of credit-card stubs and hotel bills, which 
show only the total amounts paid without any detail. 






Recommendation 





To ensure that all travel services are acquired economically, that Ontario 
government rules are followed, and that employees are reimbursed for only 
legitimate business expenditures, the Ministry should: 











e reiterate to employees the necessity of following government travel policies 
and advise staff that any exceptions to the rules will not be reimbursed; and 

e reimburse only those travel claims that are accompanied by proper 
supporting documentation. 


Ministry Response 


The Ministry agrees that travel services should be acquired economically and 
has endeavoured to contain the growth of travel costs over the past four years. 
In light of these audit findings, the Ministry will review its travel policies as they 
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relate to international travel. The Ministry will remind all staff of the importance 
of following Ministry travel policies, including the need for proper 
documentation to support their travel claims. The Ministry will also hold 


mandatory information sessions for all managers to remind them of their 
responsibility to closely review expense claims submitted by their staff to 
ensure adherence to travel policies. 





MANAGEMENT OF CONSULTING SERVICES 


During the three years ending March 31, 2003, the Ministry paid over $22 million for 
consulting services, primarily for management and technical projects. In acquiring 
consulting services, ministries must comply with Management Board of Cabinet directives 
that set out the key principles for the decisions made in planning, acquiring, and managing 
consulting services. We examined a sample of consulting assignments and found that the 
Ministry needed to improve its procedures to ensure the economical acquisition and proper 
management of consulting services, as detailed below. 


Consultants are to be hired using a competitive selection process and are to be treated in a 
fair and equitable manner. Ministries must also retain appropriate documentation for each 
assignment. We noted instances where the Ministry did not maintain an open and fair 
process for acquiring consulting services or retain appropriate documentation. For example: 


¢ One consultant the Ministry hired was selected because it was the lowest bidder at 
$50,000. Following the bid, the Ministry had some questions about the proposal and 
asked the consultant to provide additional information. As a result, the cost increased to 
$95,000 and the terms of reference in the request for proposal were different from the 
final terms agreed to in the contract. However, the Ministry did not give the other 
bidders the opportunity to resubmit their bids, which may have been lower, based on 
the final terms of reference. 


¢ In another case, the Ministry hired a consultant for $382,000. Although the Ministry 
indicated that bids were received from three other consultants, it could not provide us 
with any documentation to demonstrate that a competitive process was followed. 


Consulting contracts and payments are required to be properly monitored and controlled, 
and any increases in the ceiling prices must be justified, formally agreed to, and properly 
approved. However, we noted examples of payments made to consultants that exceeded the 
ceiling prices of the contracts by $20,000 to $116,000. These represented increases of 30% 
to 100% of the original cost. For these contracts, the terms and conditions of the 
assignments remained unchanged and we found no documentation justifying the increase 
in ceiling prices. We also found no evidence of prior approvals for the increases from the 
Deputy Minister or designate, as required by Management Board of Cabinet directives. 


For every consulting assignment, a written agreement between the Ministry and the 
consultant must be prepared as soon as possible, preferably before the assignment begins. 
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From the sample of files we examined, we noted that one consultant provided services four 
months prior to signing a contract with the Ministry, and a recent audit by the Ministry’s 
Internal Audit Services Branch also detected a number of similar exceptions. In not having a 
signed agreement in place before an assignment is undertaken, the Ministry runs the risk of 
misunderstandings with the consultant regarding their respective responsibilities and the 
scope of the work, which may impact on the successful completion of the assignment at the 
originally agreed-upon price. 


When consultants complete their assignment, the Ministry is required to prepare a written 
evaluation that includes an assessment of: the quality of the work; whether value for money 
was obtained; and the suitability of the consultant for future assignments. However, we 
found that the Ministry had not prepared the required written evaluations for any of the 
consulting assignments we reviewed. Since the four ministry divisions we audited 
independently hire consultants, such evaluations would help other proposal review teams in 
the other divisions by providing information on a consultant’s past performance. 





Recommendation 


To achieve value for money when using consulting services, the Ministry 
should ensure that: 


¢ consultants are hired through a competitive selection process and are 
treated fairly and equitably and any exceptions are adequately justified, 
documented, and approved; 


* contracts and payments are properly monitored and controlled, and any 
increases in the ceiling price are justified, formally agreed to in advance, 
and properly approved; 

* contracts outlining the key deliverables, costs, and other significant project 
details are signed before consultants begin their assignments; and 

° formal evaluations are prepared for consultants when their assignments 


are completed, and these evaluations are made available to other divisions 
for future evaluations of consulting proposals. 











Ministry Response 


The Ministry agrees that there should be value for money for consulting 
services and had taken steps prior to this review to ensure that consulting 
services are being acquired appropriately and managed effectively. 





In 2001/02, the Ministry undertook an internal audit of consulting services. That 
internal audit had noted many of the issues related to consulting services 
identified in this report. The Ministry is implementing significant improvements 
to address the findings of this report and the internal audit report. These 
improvements include updated policies and procedures, improved 
communications of ministry policies, increased controls for acquisition of 
consulting services, enhanced reporting, and customized training for staff. 
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With regard to contract management, the Ministry is working with the Shared 


Services Bureau to improve operational efficiency and compliance with 
Management Board directives. 





ONTARIO INVESTMENT SERVICE WEB SITE 


The Ministry’s Ontario Investment Service (OIS), part of the Ministry's Investment 
Division, acts as a single source of customized information, investment marketing materials, 
and other services to Ontario businesses considering expansion and to potential investors in 
Ontario. It operates a Web site that provides information of interest to potential investors, 
including: municipal profiles and industrial property information designed to assist investors 
with site selection; data on the availability of labour skills and other economic data; and 
comparative investment information on Ontario and competing jurisdictions. 


‘To carry out its responsibilities, the OIS purchased a stand-alone computer server to host its 
Web site. A consulting firm was hired under two successive contracts, which totalled 
approximately $960,000 and covered the period July 2001 to September 2003. Under the 
direction of OIS, the firm’s consultants were to provide day-to-day operational services 
related to project leadership and management of the Web site, including site design, 
database management, programming, page layout, editing, and other technical support. 
We reviewed the consultants’ job descriptions and the list of their projects for the 2002/03 
fiscal year and noted that most of the work they carried out was updating and maintaining 
aspects of the Web site. Of the 20 tasks assigned during the year, only two involved 
developing new sections for the Web site. The cost of having consultants carry out these 
tasks was significantly more than if the functions had been performed by two or three 
comparable Web co-ordinators and Web design staff at the Ministry. Salaries of similar staff 
positions in the Ministry averaged $66,000. At this rate, we estimate that hiring consultants 


instead of having ministry staff manage and maintain the OIS Web site may have cost the 
Ministry as much as $500,000 more. 











Recommendation 


To ensure that the Ministry’s requirements for the Ontario Investments Service 
Web site are met in the most economical manner, the Ministry should consider 
the costs and benefits of having the functions performed internally. 


Ministry Response 





The Ministry agrees and is taking action to ensure that the Ontario Investment 
Service (OIS) Web site is delivered in an economical manner. 





Until 2001, maintaining the OIS Web site through individual fee-for-services 
contracts was very cost effective. As a result of the move from individual 4 
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fee-for-services contracts to a full-service contract, there has been a significant 
increase in costs over the past two years. 


The Ministry has identified this as an issue to be reviewed as part of the 
program evaluation currently underway for the Investment Program. The 
program evaluation will consider available options for Web site maintenance 
and make recommendations on the most cost-effective option. 











162 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





MINISTRY OF ENTERPRISE, OPPORTUNITY 
AND INNOVATION 


3.07—Science and 
Technology 





BACKGROUND 


The mandate of the Ministry of Enterprise, Opportunity and Innovation is to foster 
competitive businesses and a prosperous economy in Ontario. Through this mandate, the 
Ministry promotes innovation, economic growth, and job creation. The Ministry's science 
and technology goals are to provide leadership for and co-ordination of related programs 
and activities government-wide and to develop strategies that foster a culture of innovation 
in the Ontario economy. 


Between April 1, 1998 and March 31, 2003, the Ministry funded several major science 
and technology programs and spent $1.3 billion. The government announced total 
program commitments (i.e., expected total funding) of $4.3 billion, as shown in the 
following table. 
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Ministry Science and Technology Commitments and Funding, 
April 1, 1998 to March 31, 2003 


Total 
Ministry 
ae larel tare | 


_ Science andTechnology Program = |= Commitments 
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Ontario Innovation Trust 1,050.0 750.0* 


Ontario Research and Development Challenge Fund 1,250.0 182.6 


Ontario Research Performance Fund 25le3 91.3 
Premier's Research Excellence Awards 45.0 


| 
: 
Premier’s Platinum Medal for Research Excellence 


Other Transfer Payments 


7 


Ontario Centres of Excellence 342.5 6WeS 


N | © 
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* The Ministry had paid the Ontario Innovation Trust $750 million as of March 31, 2003. As of March iil. 
2002, the date of the most recent information available, the Trust had disbursed $239.6 million to grant 
recipients and had more than $500 million on hand. 


Source of data: Funding Pronouncements and Public Accounts of Ontario 


There is no specific legislation related to the Ministry’s science and technology programs, as 
most programs have been established pursuant to approved Cabinet submissions. Each 
program has specific objectives and provides grants to institutions such as universities, 
colleges, hospitals, and other not-for-profit organizations to carry out scientific research and 
technology development. Furthermore, most programs require the research institution to 
obtain private-sector financial support in addition to program funding, and several 
programs use advisory boards and panels to review research proposals and recommend 


specific projects for funding. 


The Ministry’s Science and Technology Division has approximately 50 staff and, in the 
2002/03 fiscal year, spent $7.8 million on direct operating expenses. These expenses 
include payments to the Innovation Institute of Ontario, a not-for-profit corporation that 
provides administrative services for a number of ministry programs. 
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AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of the science and technology programs were to assess whether 
the Ministry had adequate procedures in place to: 


* ensure compliance with Management Board of Cabinet directives, contractual 
agreements, and ministry policies; 


* ensure that resources were managed with due regard for economy and efficiency; and 


* measure and report on the ministry’s effectiveness in fulfilling its mandate and 
identifying areas where corrective actions are required. 


The criteria used to conclude on our audit objectives were discussed with and agreed to by 
ministry management and relate to systems, policies, and procedures that the Ministry 


should have in place. 


Our audit was performed in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants, and accordingly included such tests and other procedures as we 
considered necessary in the circumstances. 


The scope of our audit, which was substantially completed by March 2003, included an 
examination of six science and technology programs that together accounted for 90% of the 
Ministry’s science and technology expenditures. We also assessed selected aspects of the 
Ministry’s other science and technology programs. Our audit included discussions with 
relevant staff, as well as a review and analysis of documentation that was made available to us 
at the Ministry and at the Innovation Institute of Ontario. Our audit also included a review 
of the activities of the Ministry’s Internal Audit Services Branch. We reviewed the Branch’s 
recent reports and incorporated any relevant concerns into our audit work. 


ACCESS TO INFORMATION 


We did not receive adequate access to all the information we requested from the Ministry— 
information that the Ministry must provide to us under section 10 of the Audit Act. For 
example, on October 23, 2002, we requested submissions to Management Board of 
Cabinet for a number of ministry programs. These submissions outline such program 
parameters as who is eligible for funding, funding criteria, and grant limits. Such 
information was vital to the performance of our audit. Also, full access to other ministry 
documentation was originally denied due to the fact that such documents were based on 
these submissions. Partial submissions were provided almost three months after we requested 
the information. Subsequently, we were allowed to read those portions of the documents 
not previously provided but were not given unfettered and timely access to all the 
documents we needed to complete our audit work. Such limitations on our access to 
information prevented us from being able to conclude on our audit objectives and complete 
this audit in a timely manner. 
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The Ministry is of the opinion that it provided the information in accordance with 
government protocols. However, we continually expressed the view that we were not 
provided with all the information needed to satisfactorily complete our audit work. 


OVERALL AUDIT CONCLUSIONS 


Given the limitations on our access to information as discussed above, we cannot provide 
reasonable assurance that our audit identified all matters that should be brought to the 
attention of the Legislative Assembly. However, our review of the Ministry’s science and 
technology spending of $1.3 billion between April 1, 1998 and March 31, 2003— 
primarily for research grants—revealed a number of significant concerns. A major concern 
was that the Ministry had committed to spending $4.3 billion without an overall strategic 
plan to set parameters and consistent policies for existing programs or to guide the 
development of new programs to meet the objectives of promoting innovation, economic 
growth, and job creation. Other significant concerns and deficiencies in the Ministry’s 
practices and procedures included the following: 


¢ Contrary to government directives on alternative service delivery, the Ontario 
Innovation Trust was set up through a trust agreement between the former Ministry of 
Energy, Science and Technology and a trust corporation without a business case 
justifying its creation. As noted in our previous annual reports (1999-2002), the 
Ministry did not implement the mandatory accountability controls to ensure that more 
than $1 billion in public funding, provided or committed to the Trust, is being spent for 
the purposes intended. The Ministry receives virtually no information from the Trust 
and does not have the required monitoring processes in place to ensure compliance with 
the Trust agreement. Therefore, neither the Legislature nor the Ministry have any 
control over spending by the Trust. 


¢ The contract between the Ministry and the Innovation Institute of Ontario (IIO) 
regarding the administration of the Ontario Research and Development Challenge 
Fund states that no documentation is to be destroyed without ministry approval. But the 
I1O did not retain panel reviewers’ assessments of research proposals and often disposed 
of these assessments without the Ministry's consent. Therefore, management could not 
demonstrate that appropriate practices were followed in the decisions made to fund 
specific projects. 


* The Division often did not have the supporting documentation that was necessary to 
properly calculate Ontario Research Performance Fund grants. We obtained 
documentation from other sources for a sample of 2001/02 recipients, recalculated the 
grants, and found that one institution was underpaid by $277,000 and another was 
overpaid by $147,000. We notified the Ministry of these errors, and the errors were 
subsequently corrected. 
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e We reviewed in detail the summary score sheets for one round of competition for the 
Premier’s Research Excellence Awards and found that the marks on the reviewers’ 
individual score sheets did not match the marks on the Ministry's summary score sheet, 
and the marks on the summary score sheet did not match the summary of scores on the 
short list of recipients recommended for funding. Such findings limit the Ministry's 
ability to demonstrate the fairness and transparency of the selection process. 


¢ The Ministry does not have adequate procedures in place to monitor potential conflicts 
of interest. For example, a review of advisory board minutes revealed occasions on which 
a conflict of interest should have been declared, but there was no indication in the 
minutes that a conflict had been declared. 


¢ In July 2000, the Ministry single-sourced the administration of the Ontario Research 
and Development Challenge Fund to the Innovation Institute of Ontario, a subsidiary 
of the Ontario Innovation Trust. According to ministry documentation, a request for 
proposals (RFP) was not issued because once an RFP is out, anyone who is eligible must 
be treated fairly in the process, and problems could arise if a bidder was not dealt with 
fairly after an investment of time and resources in the preparation of a proposal. Single- 
sourcing, especially for such a reason, contradicts the basic principles of government 
procurement. 


¢ Although the Ministry has spent hundreds of millions of dollars on science and 
technology research, it has made little effort to ensure that intellectual property rights 
arising from funded research ultimately benefit the province. 


¢ The Ministry did not have a process in place for measuring and reporting on the 
achievement of its overall goals of promoting innovation, economic growth, and job 
creation. The Ministry measured its performance in terms of growth in the value of 
non-government-sponsored research over the next five years. This measure is a future- 
oriented target or benchmark and does not reflect the ongoing impact of the Ministry's 
expenditures to date. 


DETAILED AUDIT OBSERVATIONS 


COMPLIANCE WITH GOVERNMENT AND 
MINISTRY POLICIES 


Governance and Accountability 


The Management Board of Cabinet directive on Transfer Payment Accountability requires 
the Ministry to establish an effective framework for the prudent management of provincial 
transfer-payment funds. The mandatory requirements include defining expectations, 
signing agreements, ongoing monitoring, reporting periodically, and taking corrective 
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action when necessary. We reviewed the governance and accountability processes in place 
for the Ontario Innovation Trust and for the Ontario Research and Development 


Challenge Fund. 


ONTARIO INNOVATION TRUST—GOVERNANCE AND ACCOUNTABILITY 


The Ontario Innovation Trust (Trust) was created in March 1999 to help support the 
capital costs of research performed in Ontario universities, community colleges, hospitals, 
and research institutions. The Trust was intended to match federal grants from the Canada 
Foundation for Innovation and to complement the Ontario Research and Development 
Challenge Fund, which would fund operating costs for research. A governing board was 
appointed to oversee the Trust, and in 2000, a subsidiary, the Innovation Institute of 
Ontario, was created to administer Trust grants to research institutions. The Ministry has 
committed to pay the Trust more than $1 billion, and as of March 31, 2003, had paid 
$750 million. The Trust had more than $500 million on hand as of March 31, 2002—the 


date of the most recent information available. 


The ‘Trust was created by an agreement between the former Minister of Energy, Science 
and Technology and a private-sector corporation that was designated as the Trustee. But 
contrary to government directives on alternative service delivery, we were informed that 
there was no submission to Cabinet and no business case to justify the Trust’s creation. Also, 
as noted in our previous annual reports (1999-2002), contrary to the Management Board 
of Cabinet directive on Transfer Payment Accountability, the Ministry did not enter into an 
agreement with the Trust to define provincial expectations or outline how the Trust would 


be held accountable for its spending of public funds. 


The Transfer Payment Accountability directive specifies that the Ministry must routinely 
obtain and review information on the status of recipient eligibility and performance and 
that the Ministry is responsible for identifying non-compliance with agreements and 
initiating corrective action. However, we observed that the Ministry receives virtually no 
information from the Trust and does not have the required monitoring process in place to 
ensure compliance with the trust agreement. For instance, the trust agreement specifies 
maximum amounts that can be provided for an eligible project, but ministry management 
cannot monitor whether eligible projects have been split into several components to 
circumvent these funding limits. Also, without a process for verifying eligibility, the Ministry 
has no assurance that the Trust is spending public funds in compliance with the trust 
agreement. 


The Trust is not required to report to the Ministry or through the Minister to the 
Legislature. While the Trust has voluntarily posted selected information on its public Web 
site, the Trust’s first annual report was not released until three years after the Trust’s creation. 
In addition, although ministry staff indicated that the Trust’s Board had hired a consultant 
to report on whether the Trust implemented best practices in accountability, the Ministry 
did not know whether the review actually took place or whether a report was ever issued. 
We also requested, but did not receive, the consultant’s report. 
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Having representation on the Trust’s Board could help the Ministry monitor the Trust’s 
activities. The trust agreement states that of the seven members of the Trust’s Board, three 
shall be appointed by the Lieutenant Governor in Council on the Minister’s 
recommendation. Although the trust agreement stipulates that a member whose term has 
expired shall remain in office until a successor is appointed, the wording used in the 
Lieutenant Governor’s appointments states that these appointments are “not to exceed” the 
time period stipulated. Yet one government appointment expired in May 2001 and another 
in May 2002, and the government has not yet appointed replacements. In addition, none 
of the three government employees appointed to the Board by the Lieutenant Governor 
were ministry staff. We are concerned that even this basic oversight provision to monitor the 


planned spending of $1 billion in public funds is not being achieved. 


In summary, neither the Legislature nor the Ministry has control over the spending of 


taxpayers’ funds by the ‘Trust. 





Recommendation 


To ensure the effective oversight of the Ontario Innovation Trust’s spending of 
potentially more than $1 billion in public funds, the Ministry should: 


e negotiate an agreement with the Trust to establish proper governance and 
accountability arrangements; 


e implement procedures for routinely obtaining and reviewing information on 
the status of recipient eligibility and ongoing results; 


¢ implement procedures for identifying areas of non-compliance and 
initiating corrective action where required; and 


¢ ensure that all government Board appointments are up to date. 


Ministry Response 


The Ministry will make its best efforts to ensure accountability. However, it 
should be noted that the Ministry does not have the authority to impose an 
agreement on the Ontario Innovation Trust (Trust), although the Ministry could 
take steps to negotiate an agreement as suggested. 


In order to try and ensure that the Trust is in compliance with the 
accountability framework established in the March 31, 1999 Trust Agreement, 
the Ministry will: 

e formally request a quarterly update from the Trust to confirm recipient 
eligibility and results of investments beginning in the fourth quarter of the 
2003/04 fiscal year; 

work with the Trust to put in place process tools to identify and address 
any non-compliance matters; and 


ensure that the government appointments are brought up to date as quickly 
as possible, recognizing that they are Order-in-Council appointments. 
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ONTARIO RESEARCH AND DEVELOPMENT CHALLENGE FUND— 
GOVERNANCE AND ACCOUNTABILITY 

In 1997, the Ontario Research and Development Challenge Fund (Challenge Fund) was 
established as a 10-year program to promote research of interest to the private sector and 
encourage collaboration between research institutions and the private sector. The Challenge 
Fund was also designed to improve Ontario’s ability to attract and keep researchers in the 
province and to help Ontario universities compete for federal research funding. As of 
March 2003, the Minister had approved more than 100 Challenge Fund projects, for a 
total ministry commitment of $435 million. We were advised that the sponsoring research 
institutions and private-sector organizations had made additional funding commitments of 
$437 million and $510 million, respectively. 


At the Challenge Fund’s inception in 1997, a Challenge Fund Board was created as an 
independent body to provide advice and recommendations to the Ontario government on 
individual research and development proposals. The Ontario government was represented 
on that Board by the Ministries of Energy, Science and Technology; Economic 
Development and Trade; Training, Colleges and Universities; Agriculture, Food and Rural 
Affairs; and Finance. A 1997 submission to Cabinet established the former Ministry of 
Economic Development, Trade and Tourism as the lead ministry responsible for 
administering the Challenge Fund and being accountable to the Legislature. In 

August 1999, a Memorandum of Understanding (MOU) between the Ministries and the 
Board outlined their respective roles and responsibilities. This MOU was to be formally 
reviewed every two years. 


The first MOU review should have been completed in August 2001 but had not been 
done as of March 2003. Significant changes have occurred in the Challenge Fund that 
would warrant updating the MOU. For example, a program objective to match federal 
grants for capital infrastructure was dropped with the creation of the Ontario Innovation 
Trust. Also, since November 2000, the monitoring of grants and other responsibilities of the 


Challenge Fund have been delegated to the Innovation Institute of Ontario, a subsidiary of 
the Trust. 


The MOU permits the Challenge Fund Board’s Chair to appoint special advisory 
committees to assist in evaluating proposals for research funding. To date, a non-medical 
panel and a medical panel have been established to assess proposals and provide summaries 
to the Board for decision-making. But there are no terms of reference outlining key features 
of such panels, including the panels’ role, who can serve on them, the length of members’ 
terms, and the panels’ policies, procedures, and processes. Furthermore, the MOU does not 
establish performance targets or a lead ministry for the program. 
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Recommendation 





To ensure that the required accountability mechanisms are in place for the 
management of the Ontario Research and Development Challenge Fund 
(Challenge Fund), the Ministry should: 





e update the Memorandum of Understanding between the Challenge Fund 
and the ministries to outline the responsibilities of the Challenge Fund 
Board and special advisory committees and to reflect current program 
objectives and suitable performance measures; and 


e ensure that primary and direct oversight responsibility for the Challenge 
Fund rests with a lead ministry. 








Ministry Response 


The Ministry agrees that it must update the Memorandum of Understanding 
(MOU) to reflect the changes to the Ontario Research and Development 
Challenge Fund since the program was established. The update will be 
completed by December 31, 2003, and will include: the role of the Challenge 
Fund Board and ongoing special advisory committees; program objectives; 
and performance measures. The changes to the MOU will also incorporate the 
best-practice advice that the Ministry has recently received from an 
International Review Panel and will reflect the Ministry’s role as lead ministry. 


Project Selection 


The Ministry is ultimately responsible for ensuring that the selection of potential research 


and development projects meets the program’s specific objectives and specified eligibility 
requirements. To ensure that funds are provided for the most deserving proposals, a fair, 
transparent selection process must be in place. We examined the selection process and 
supporting documentation for the Ontario Research and Development Challenge Fund, 
the Ontario Research Performance Fund, the Premier’s Research Excellence Awards, and 
the Premier’s Platinum Medal for Research Excellence. 


ONTARIO RESEARCH AND DEVELOPMENT CHALLENGE FUND— 
PROJECT SELECTION 

Since November 2000, the Innovation Institute of Ontario (IIO) has administered the 
Ontario Research and Development Challenge Fund (Challenge Fund), which was 
formerly administered by the Ministry. The IIO’s responsibility is to review and analyze 
proposals for Challenge Fund grants to ensure that the documentation submitted is 
complete and that proposals meet program eligibility criteria. The IIO then is to prepare 
project summaries for use by the Challenge Fund Board. In conjunction with the Board, 
the IIO is expected to determine which proposals require expert assessments and the type of 
panel review required. Panels are then to be set up to review proposals, and the IO is to 





Science and Technology 171 


| VFM Section 3.07 | 








| VFM Section 3.07 | 





summarize these panels’ recommendations and forward the summaries to the Board for 
further consideration. The Board’s recommendations are then to be forwarded to the five 


Ministers for final approval. 


We reviewed the selection process and found that procedures were not in place to ensure 
that only eligible projects were selected and that the process was fair and transparent. Some 
of our key concerns are as follows: 


¢ The contract between the Ministry and the IIO states that no documentation is to be 
destroyed without the Ministry’s prior written approval. Documentation, such as 
evaluations by individual panel members, is to be retained to help ensure that a fair and 
transparent evaluation process was followed, that conflict-of-interest guidelines were 
observed, and that the projects recommended for approval were ranked higher than 
those recommended for rejection. But the IIO did not routinely keep on file panel 
reviewers’ individual assessments of proposals, retaining only the summaries. We found 
no evidence that the Ministry had given written consent to dispose of these documents. 
Asa result, the IO could not demonstrate that panel members’ evaluations were 


consistent with the summary recommendations prepared by the IIO and presented to 
the Board. 


* To stimulate job creation and economic growth, Challenge Fund policy requires that a 
third of any project's costs be received from or committed by private-sector partners. 
These commitments cannot be made more than six months before the deadline for 
submitting an application. Contributions committed before that time are not eligible 
because they do not represent newly created partnerships or incremental funding. We 
found that the Ministry and the IIO did not always ensure compliance with these 
eligibility requirements. For example, an applicant applying for a grant with a private- 
sector partner was awarded $9.3 million from the Challenge Fund. Contrary to Fund 
policy, the project had started and the private-sector partner had made commitments to 
the project a year before applying for the grant. This project did not create new 
partnerships and did not generate incremental or new private-sector funding. 


¢ The Memorandum of Understanding (MOU) between the Challenge Fund Board 
and the five ministers responsible for the program requires the Board to provide written 
recommendations of each proposal to be considered to all ministers who are a party to 
the agreement. The Ministry requires the Challenge Fund Board to provide written 
recommendations regarding each proposal to the five ministers within 70 days of the 
application deadline. We reviewed the timeliness of the process for the last several 
competitions and found that the process was not completed in a timely manner. The 
ITO could not provide evidence of when recommendations were submitted to the 
Minister, and there are no guidelines for the timeliness of the Minister’s approval. 
Overall, the time taken from the application deadline to ministerial approval ranged 
from 89 to 273 days. In addition, the Ministry could not provide any evidence that 
written recommendations were provided to, and approval was received from, any of the 
other ministers who were party to the agreement, as required by the MOU. 
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Recommendation 


To ensure that the Ontario Research and Development Challenge Fund 
selection process is timely, fair, and transparent, and that adequate procedures 
are in place to assess project eligibility, the Ministry should: 


¢ require the Innovation Institute of Ontario to retain all relevant 
documentation; 


e implement procedures for periodically verifying eligibility and ensuring that 
any exceptions to program eligibility criteria are weil supported; 


e ensure that all applicable ministers are apprised of the Board’s 
recommendations or, if appropriate, obtain a delegation of authority for the 
Minister of Enterprise, Opportunity and Innovation to approve projects for 
funding on behalf of all ministers who are party to the agreement; and 


¢ ensure that applications for research funding are reviewed within the 
specified time frame and that recommendations are made to the required 
ministers on a timely basis. 


Ministry Response 


The Ministry will work with the Innovation Institute of Ontario (lO) to ensure 
appropriate procedures are being implemented for retaining documents, 
verifying eligibility, and supporting any exceptions. The service agreement with 
the IIO requires the IIO to retain all relevant documentation, and the Ministry 
will strictly enforce this requirement. 





The Ministry agrees that there are administrative efficiencies to be achieved 
with delegated authority to the Minister of Enterprise, Opportunity and 
Innovation. The Ministry will pursue this approach with its partner ministries by 
December 31, 2003. 


The Ministry is committed to finding ways to ensure that applications are 
reviewed in a timely manner and that recommendations are made on a timely 
basis. 





ONTARIO RESEARCH PERFORMANCE FUND—PROJECT SELECTION 


The Ontario Research Performance Fund (Performance Fund) was introduced in the 
2000 Ontario Budget to provide overhead funding for institutions conducting Ontario- 
government-sponsored research. For eligible research projects, the Performance Fund 
contributes to overhead costs (such as heat and hydro) at a rate calculated at 40% of the 
project’s direct costs. The rate is reduced when overhead support is received from other 
ministries. During the 2001/02 fiscal year, the Performance Fund made payments for 218 
research projects. In its first two years, the Performance Fund was to spend $30 million 
annually. If the total overhead funding for all eligible recipients exceeded this amount, each 
recipient’s funding was to be reduced to ensure that the total final payments for the year did 
not exceed $30 million. 
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A July 2000 Management Board submission outlined the Performance Fund’s 
administrative framework and noted that those institutions designated as eligible research 
institutions in a provincial tax bulletin would be deemed eligible for program funding. 
Institutions not listed in that bulletin could apply to the Ministry for an assessment of their 
eligibility status. However, we found that the Ministry did not have documentation to 
demonstrate the process or rationale used to approve the eligibility of 41 new recipients that 
were not listed in the tax bulletin. In the 2001/02 fiscal year, the Performance Fund paid 
these recipients $8.7 million. 


Eligibility is also contingent on a signed confirmation letter from the lead ministry that 
provided the initial research funding. This letter confirms the amount of the initial grant for 
direct costs. This amount is used to calculate the Performance Fund’s overhead grant. Since 
confirmation letters were not on file for a number of projects, the Ministry often did not 
have sufficient documented evidence regarding the amounts used to calculate the 
Performance Fund grants. We recalculated the grants for a sample of 2001/02 recipients 
and found that one institution had been underpaid by $277,000 and another had been 
overpaid by $147,000. We notified the Ministry of these errors, which were subsequently 


corrected. 


Performance Fund grants are provided to reimburse overhead costs for the preceding year. 
Under the terms of the contract between the institutions and the Ministry, institutions must 
submit a performance report outlining the use of grant monies in the year in which the 
grant was received. However, because the funds are to be used to reimburse the previous 
year’s costs, not the current year’s, which could be significantly different than the prior year’s 
overhead costs, this reporting requirement does not meet its intended purpose. 


Since there is no deadline for eligible recipients to submit funding requests, submissions can 
be received after all of the program’s annual funding has been paid out. In 2000/01 and 
2001/02, late submissions totalling $590,000 and $116,000, respectively, were paid in the 
subsequent fiscal years. Since no adjustments were made to the amounts paid to other 
recipients, these payments resulted in overpayments or payments in excess of the total 
annual program funding for that particular year. At the time of our audit, program 


payments for the 2002/03 fiscal year, including any adjustments or overpayments, had not 
been finalized. 


Recommendation 


To help ensure that recipients of Ontario Research Performance Fund grants 
meet eligibility criteria and are paid the proper amounts, the Ministry should: 


ensure that all new recipients meet program eligibility requirements; 


ensure that signed confirmation letters are on file verifying the amount of 
grants provided by other ministries to eligible recipients; 

implement procedures for verifying that grant amounts are calculated 
accurately; and 
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e establish a deadline for submissions and for finalizing annual payments 
under the program. 


Ministry Response 


The Ministry agrees and is in the process of updating its procedures for the 
Ontario Research Performance Fund to address these issues. Specifically: 


¢ The Ministry has put in place an administrative practice to require lead 
ministries to be accountable for ensuring that all new recipients meet 
program eligibility requirements. 
Copies of all signed confirmation letters verifying the amounts of grants 
provided by other ministries will be kept on file at the Ministry in a new 
database tracking-and-management system that is currently being 
implemented. 


The Ministry has implemented new procedures for verifying that grant 
amounts are calculated accurately. 
The Science and Technology Division will undertake an internal review of 


its submission and payment processing system prior to the end of the 
2003/04 fiscal year. 
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PREMIER’S RESEARCH EXCELLENCE AWARDS—PROJECT SELECTION 


The Premier’s Research Excellence Awards (PREA) program was established in 1998 to 
attract, develop, and keep talented graduate students and researchers in Ontario. The 
awards are made on a competitive basis to principal researchers through their research 
institutions. PREA’s nine-member volunteer advisory board reviews all applications and 
makes award recommendations to the Minister. While each recipient is eligible for up to 
$100,000 in program funding, an additional 50% funding from either a research 
institution or a private-sector partner is required. As of March 31, 2003, approximately 
$45 million had been awarded to 438 recipients. 


Potential award recipients must submit a standard application, letters of reference, and 
letters guaranteeing the 50% matching private-sector contribution. Ministry staff distribute 
the applications to pairs of Board members for review. Each pair jointly recommends 10 
applications for further consideration by the entire Board. The Board Chair reviews all 
applications, and especially complex applications may also undergo an external expert peer 
review. The score sheets and ratings from the review pairs, external expert peer reviewers, 
and the Board Chair are forwarded to the Ministry. The Ministry is to summarize the score 
sheets and prepare a short list of applicants. The PREA Board considers the Ministry's short 
list, determines final recommendations, and forwards these to the Minister for approval. 


We reviewed the PREA selection process and found that the Ministry had not maintained 
sufficient documentation to support the selection decisions. For example, the Ministry 
retained score sheets for successful applicants only. Consequently, it was not possible to verify 
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whether the most deserving applicants, as assessed by the review panel, were those 
recommended for awards. In addition, the reasons for rejecting applicants were not 
documented in the available Board minutes, and as of March 31, 2003, the minutes 
regarding all 2002 awards had not yet been prepared. Furthermore, the Ministry was 
unable to provide the summary score sheets for five of the seven competitions completed to 
date. We reviewed in detail the Ministry's summary score sheets for one of the two 
competitions that were available and noted that: 


* one out of every four individual score sheets reviewed included only an overall score, as 
opposed to a separate score for each of the selection criteria; 


e the overall marks on the individual score sheets did not match the marks on the 
Ministry’s summary score sheet; and 


¢ the scores on the summary score sheet did not match with the summary of scores on the 


Ministry’s short list. 


The selection criteria were also not clearly communicated to potential applicants. Award 
recipients are selected based on four criteria: the applicant’s publication record, the ability to 
attract financial support from other sources, letters of reference, and the merits of the 
proposed research plan. The Ministry’s Internal Audit Services Branch surveyed grant 
recipients following the first round of competition and found that the selection process and 
program criteria were unclear. We reviewed the more recent competitions and noted that 
the evaluation criteria, scoring process, and weightings for each criteria were still not 
explicitly communicated to applicants. Guidelines are provided to reviewers on scoring and 
weighting for each criterion, but the guidelines state that reviewers have the discretion to 
change the criteria’s weightings for the final score. Ministry staff informed us that the 
assigned scores were only a guide and that reviewers’ comments had a greater impact on the 
final recommendations. In such an environment there is no assurance that the scoring 
criteria for all proposals are comparable. 


There were other inconsistencies in the evaluation process. For instance, a different rating 
system was available for new researchers. The guidelines state that given new researchers’ 
lack of granting history and independent publications, more weighting should be placed on 
letters of support and on the research plan. The Board also recommended that all 
applications by new researchers be externally reviewed. There is merit in using an alternative 
evaluation method for these applicants; however, the practice was not consistently applied. 
We identified examples where each of the four criteria were given equal weighting, and 
others where applicants were evaluated in only three of the four categories. In addition, only 
two-thirds of the applications by new researchers were externally reviewed. 
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Recommendation 


To provide assurance that a fair and transparent selection process is followed 
and that due diligence is demonstrated when assessing proposals for the 
Premier’s Research Excellence Awards, the Ministry should ensure that: 


all selection documents—including proposals, individual score sheets, 
summary score sheets, and written recommendations—are kept on file for 
a specified retention period; 


all individual and summary score sheets are reviewed for accuracy; and 


the selection process and evaluation criteria are explicitly stated to 
potential applicants and applied consistently. 


Ministry Response 


The Ministry agrees and will review, update, and document its procedures for 
Premier's Research Excellence Awards (PREAs) to address these issues. 
Specifically: 
e¢ All relevant selection documents will be kept on file for a specified 
retention period. 
The Ministry will work with the PREA Board to review the evaluation 


methods to improve accuracy and consistency prior to the spring 2004 
round of the PREA program. 





PREMIER’S PLATINUM MEDAL FOR RESEARCH EXCELLENCE— 
PROJECT SELECTION 


The Premier’s Platinum Medal for Research Excellence (Premier’s Platinum Medal) was 
created in 2001 to keep outstanding senior researchers in the province. The award is 
intended for researchers at the mid-career level who have carried out a substantial amount 
of their work in Ontario, who have an international reputation for excellence, and whose 
body of research has contributed significantly to furthering knowledge in their field. The 
program is slated to cost the province $10 million over six years. The award includes a 
platinum medal, a citation, and a $1 million cash prize to support the winner’s research at 
an eligible Ontario research institution, either by covering direct research costs or by 
endowing a scholarship, research fellowship, or chair in the recipient's name. 


The Ministry’s June 2001 Management Board of Cabinet submission established the 
Premier’s Platinum Medal and assigned the responsibility for overseeing the new program to 
the Premier’s Research Excellence Awards (PREA) Advisory Board. The Cabinet submission 
indicated that the Ministry would amend the Memorandum of Understanding (MOU) 
between the government and the PREA Board to outline terms of reference specific to the 
new program, the Board’s mandate and responsibilities with regard to the new program, 
and the performance measures by which the new program’s success would be evaluated. As 


of March 31, 2003, no changes had been made to the MOU. 
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The potential recipients of the Premier’s Platinum Medal were nominated by their peers, 
and the selection process began by establishing a six-member subcommittee drawn from the 
PREA Advisory Board and the Ontario Science and Innovation Council, an independent 
body created to provide the government with long-term strategic advice on science, 
technology, and innovation. The subcommittee conducted an initial assessment of 
candidates to create a short list of seven nominees. The short list was forwarded to six 
international reviewers for their opinions and recommendations. After reviewing the 
international reviewers’ assessments, the PREA Advisory Board recommended two nominees 
to the Minister for approval. At the time of our audit, although payments had not yet been 
made, the first two award winners had been announced. 


Given the profile and prestige of the Premier’s Platinum Medal, a fair and transparent 
selection process is essential. However, we noted that one candidate’s nomination letter was 
dated 19 days after the competition’s October 31, 2001, nominations deadline. The 
Ministry had not stamped the nomination package or recorded when the nomination was 
received. Also, adequate documentation to support the selection of the seven shortlisted 
candidates was not maintained by the Ministry. Subcommittee documentation was available 
from only four of the six members. In addition, most individual nominee score sheets were 
not on file. For many of the individual score sheets that were on file, there was no indication 
who had performed the evaluation. According to the subcommittee’s recommendation 
letter to the Minister, 25 people were nominated for the award. However, one 
subcommittee member had made an attempt to categorize all the nominations reviewed, yet 
that categorization included only 14 of the 25 candidates. Without complete score sheets 
from all reviewers for each nominee, there is no documented assurance that the review 


process had been conducted in a fair and transparent manner. 





Recommendation 






To ensure that a fair and transparent selection process is in place for selecting 
recipients of the Premier’s Platinum Medal for Research Excellence, the 
Ministry should: 







e update the Memorandum of Understanding with the Advisory Board to 
reflect the Board’s responsibility for the program; and 


e retain all documentation necessary to adequately support the eligibility and 
selection of each recipient of the Premier’s Platinum Medal for Research 
Excellence. 











Ministry Response 







The Ministry agrees and is currently updating the Memorandum of 
Understanding to outline the terms of reference for the program, the Board’s 
mandate and responsibilities, and the performance measures. 
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All documentation necessary to adequately support the eligibility and selection 


of each recipient of the Premier’s Platinum Medal for Research Excellence will 
be retained. 





Program Monitoring 


The Management Board of Cabinet directive on Accountability states that once 
expectations are clearly defined, effective accountability requires that there be reporting on 
and monitoring of performance in relation to those expectations. We reviewed the Ministry's 
monitoring efforts for three major grant programs: the Ontario Research and Development 
Challenge Fund, the Ontario Centres of Excellence, and the Premier's Research Excellence 
Awards. We also examined the Ministry's monitoring of potential conflicts of interest. 


MONITORING THE ONTARIO RESEARCH AND DEVELOPMENT 
CHALLENGE FUND GRANTS 


The Transfer Payment Accountability directive of the Management Board of Cabinet states 
that program managers must routinely obtain and review information on the status of 
recipient eligibility and performance. Program managers must also ensure that all required 
reports are received when due, and they must review and analyze these reports on a timely 
basis. In addition, program managers are responsible for identifying non-compliance with 
agreements and any failure by recipients to demonstrate continued eligibility. The 
Management Board of Cabinet’s Accountability Directive states that the obligation to 
answer for results and for how responsibilities are discharged cannot be delegated to other 
parties. 


The Ontario Research and Development Challenge Fund (Challenge Fund) has an 
established monitoring process that requires research institutions to submit quarterly 
disbursement requests, annual progress reports, audited reports at various stages of the 
project, and a final report when the project is completed. Research institutions are to submit 
the required reports to the Innovation Institute of Ontario (IIO), which administers the 
Challenge Fund. We reviewed this monitoring process and noted the following: 


¢ In accordance with the agreement between the Minister and the research institution, an 
institution that has not requested a quarterly disbursement for the last six months is 
deemed to have abandoned the project. At the time of our audit, the ITO was 
administering approximately 60 projects that were listed as active. But almost half these 
projects had not received funding for more than six months. More than 20 projects had 
not received funding from the program in more than a year, and one of these projects 
had not received a payment in more than three years. IIO staff told us that some of 
these projects were delayed, others had not submitted the required quarterly reports or 
had submitted information that had proven to be problematic, and the status of seven 
of the projects was not known. These levels of inactivity warranted a more thorough 
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follow-up, which could result in termination of the agreement between the Minister 
and the research institution. Funds could then be allocated to other projects that could 
help the Ministry meet its objectives in a more timely manner. 


Program staff use annual progress reports to determine whether milestones and targets 
agreed to in the project contract have been met and whether the project is on schedule. 
From our sample of active projects, we found that only about 10% had submitted the 
required annual report on time. Most annual reports were submitted late—missing the 
deadline by an average of eight months—and 30% of these projects had never 
submitted an annual report. The I[O did not follow up to ensure that all annual reports 
were submitted on time. Periodic reporting and follow-up are essential to help identify, 
for corrective action, situations where funds are not being used for the purposes 
intended. 


The Ministry has not established an overall policy regarding when audited reports, 
outlining where grant monies were spent, are required. Consequently, the requirement 
for audited reporting was inconsistent. The frequency of reporting was loosely based on 
the value of the approved grant. Some projects were required to submit audited reports 
annually, others at the end of a project, and still others at other specified times. In 
addition, there are no clear guidelines on the audited report’s required contents. Some 
projects submit financial statements, while others submit only a statement of expenses. 
Based on the scheduled dates when audited reports were to be submitted, only one of 
the audited reports we tested had been submitted on time. Almost half had been 
submitted late, and the other half had not been submitted at all. On average, audited 
reports were overdue or submitted late by almost one year; some reports were two years 
overdue and had still not been received. According to the contract between the 
Ministry and the IIO, the Ministry must review and examine financial reports from 
recipients. There was little evidence on file that the Ministry had formally reviewed the 
audited reports that had been received. 


When a project is completed, program policy requires recipient institutions to submit a 
final report summarizing the project’s accomplishments in relation to the milestones and 
deliverables agreed to in the contract. At the time of our review, 15 projects had been 
completed, but only three of these projects had submitted their final reports on a timely 
basis. The other 12 final reports either had been submitted late or remained 
outstanding, | 


The lack of follow-up to ensure that the required reports were received indicates that the 
I1O and the Ministry cannot demonstrate that project milestones have been achieved or 
that program funds have been used for the purposes intended. 
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Recommendation 


To ensure that Ontario Research and Development Challenge Fund (Challenge 
Fund) grants are used for the purposes intended and that project performance 
is reported on and monitored, the Ministry should: 


e review for continued eligibility all projects that have not received payments 
from the Challenge Fund in the previous six months, and implement an 
ongoing process for identifying and following up on such projects; 


e establish an overall policy regarding when audited reports are required, 
implement clear guidelines on the form and content of these reports, and 
ensure that quarterly, annual, audited, and final project reports are received 
when due; and 

¢ ona timely basis review and analyze all reports received to ensure that 
projects remain eligible, to determine whether milestones have been met, 
and to assess whether performance has been satisfactory. 








Ministry Response 


The Ministry agrees with the recommendation and will work with the 
Innovation Institute of Ontario (IIO) to update policies and procedures for the 
monitoring of the Ontario Research and Development Challenge Fund 
(Challenge Fund) program, as outlined. Specifically: 


e The Ministry will ask the IIO to provide, by December 31, 2003, a project 
review and recommendations on projects that have not been funded in the 
past six months and will work with the IIO to develop a quarterly reporting 
process. 

e Areview of procedures and of the form and content of reports will be 
undertaken by March 31, 2004, to ensure that quarterly, annual, audited, and 
final project reports are received in a timely manner. 

¢ Ministry staff will review and analyze reports received from its service 

provider to assess performance. 




















MONITORING THE ONTARIO CENTRES OF EXCELLENCE 


The Ontario Centres of Excellence (OCE) program was established in 1987 to provide 
industry with greater access to leading edge ideas and to help transform such knowledge 
into new products and services. There are currently four Centres of Excellence: Materials 
and Manufacturing Ontario, Communications and Information Technology Ontario, 
Photonics Research Ontario, and the Centre for Research in Earth and Space Technology. 
Annual funding for all four centres combined is approximately $33 million. 


The Ministry has provided the OCE program with more than $500 million since the 
program was established in 1987. To ensure that these funds are used to achieve program 
objectives, Management Board of Cabinet directives require the Ministry to implement an 
effective monitoring process. Such a process would require the Ministry to receive from 
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each of the four centres performance reports demonstrating the prudent use of public 


resources in compliance with defined expectations. The Ministry requires the centres to 


submit annual operating plans, audited financial statements, and reports from the president. 
We reviewed the Ministry’s monitoring process and noted the following: 





Recommendation 


To help ensure that an adequate monitoring process is in place to demonstrate 
that the Ontario Centres of Excellence use public resources prudently and in 
compliance with defined performance expectations, the Ministry should: 


The Ministry is required to review and approve an annual operating plan for each 
centre that is to be submitted 60 days before the fiscal year starts. The operating plan is 
to include a comprehensive budget, general objectives, strategies for achieving those 
objectives, a list of research projects to be funded, and commentary on initiatives to be 
undertaken throughout the year. We reviewed the operating plans submitted for the 
2002/03 fiscal year and found that although three of the four plans had been 
submitted late, all four were eventually submitted; all the major requirements were 
included in each plan; and the Ministry had reviewed and approved all four before the 
fiscal year began. 


Within 90 days after each fiscal year-end, the four Ontario Centres of Excellence are 
required to submit audited financial statements and a report from the centre’s auditor 
reconciling ministry funding with actual spending. Although the receipt dates were 
unknown, at the time of our audit the Ministry had received audited financial 
statements from all four centres for the 2001/02 fiscal year. 


Within 90 days after each fiscal year-end, each centre’s president is required to submit 
for the Ministry's review and approval a report detailing the program’s status, the centre’s 
affairs and progress, all sources of funds and spending, and performance measures of the 
centre's activities. Ministry staff informed us that none of the centres had ever submitted 
such a president's report, but that all had submitted annual reports that contained some 
of this report’s required elements. However, at the time of our audit (in March 2003), 
none of the centres had submitted an annual report for the 2001/02 fiscal year. The 
most recent annual reports on file at the Ministry were for the 2000/2001 fiscal year, 
and some of the financial information in these annual reports did not match other 
information received by the Ministry. There was no evidence on file that the Ministry 
had followed up on these discrepancies. 





implement a process for tracking the receipt of all required monitoring 
reports, and follow up on any outstanding reports in a timely manner; and 
adequately review all reports received and reconcile the annual reports’ 
information with that contained in the audited financial statements to 
ensure that reported information is accurate and complete. | 
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Ministry Response 





The Ministry agrees and will ensure that appropriate monitoring and reporting 
is put in place for the Ontario Centres of Excellence Program. 


The Ministry is currently implementing a new governance structure for the 
Centres through the Ontario Centres of Excellence Inc., a not-for-profit 
corporation that will be under contract to the Ministry to manage the Centres. 
The contract will set out performance measures and requirements for 
accountability and good governance. This contract will be completed by 
March 31, 2004. 





MONITORING THE PREMIER’S RESEARCH EXCELLENCE AWARDS 


Institutions whose researchers receive grants from the Premier’s Research Excellence Awards 
(PREA) program are required, by the terms of their contracts with the province, to submit 
annual financial and performance reports to the Ministry within six months after fiscal year- 
end. This reporting requires the institution to set up and maintain a separate accounting for 
each grant, allowing it to track project spending and to record any other financial support 
received. Performance reports should include the amount of private-sector money 
contributed or committed; the number of researchers supported by the award; and the 
number of publications, patents, and licences resulting from award recipients’ research. The 
Ministry is responsible for tracking performance measures and is required to consolidate the 
performance information into an annual report to the PREA Board for review and 
transmission to the Minister. 


By October 1, 2002, a third of the PREA recipients we sampled had not submitted the 
required financial reports, and more than half had not submitted performance reports, for 
the previous fiscal year. Some reports were outstanding by as much as three years. Most of 
the reports that had been received were submitted late. The Ministry did not track 
submissions from award recipients to ensure that all required reports were received on time, 
nor was there evidence of adequate follow-up action by the Ministry. 


Financial reports are necessary to allow the Ministry to verify that recipients are complying 
with the terms of their contract—for example, by ensuring that funds are being used for the 
purposes intended and that the required private-sector contributions have been received. 
But the Ministry did not adequately review the financial reports submitted by recipients, 
and the accuracy of many such reports was questionable. For several of the reports we 
reviewed, the cumulative value of the province’s contribution as reported by the recipients 
in successive periods exceeded the maximum provincial contribution under the funding 
agreement. The Ministry had not identified these discrepancies for follow-up. 


Performance reports help enable the Ministry to assess whether a program ts accomplishing 
its goals of attracting, developing, and keeping talented graduate students and researchers in 
Ontario. Recipient institutions are responsible for tracking and reporting specific 
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performance measures, including the number of researchers supported by the award and 
the number of students in the research group ultimately employed in Ontario industries. 
The Ministry is responsible for consolidating performance information (such as the 
movement of researchers into and out of the province) and for reporting annually on the 
program’s results. However, in addition to not ensuring that all required performance 
reports were received, the Ministry had not consolidated the information that was 


submitted and had not reported to the PREA Board and the Minister as required. 





em - 


Recommendation 


To help ensure that the Premier’s Research Excellence Awards program meets 
its objectives of attracting, developing, and keeping talented graduate students 
and researchers in Ontario and that funds are spent appropriately, the Ministry — 
should: 


¢ ensure that all required financial and performance reports are received ona 
timely basis; 

e verify that funds are being spent for the purposes intended, that the 
information submitted is accurate, and that project targets and milestones 
are being met; and 


* analyze and consolidate the performance information reported by recipient 
institutions to assess the program’s accomplishments, and report this 
information annually to the program’s board and to the Minister as 
required. 


Ministry Response 


The Ministry agrees and will review current practices to verify that funds are 
being used for the purpose intended and that appropriate information on 
performance results for the Premier’s Research Excellence Awards program is 
being received, analyzed, consolidated, and reported on a timely basis. oo 











MONITORING POTENTIAL CONFLICTS OF INTEREST 


To ensure that government activities are conducted in an open, fair, and transparent 
manner, the Management Board of Cabinet has issued two directives that address potential 
conflict-of-interest situations for public servants and for individuals appointed by the 
government—the Conflict of Interest and Post-service Directive for Public Servants and 
Public Officials, and the Government Appointees directive. A conflict of interest occurs 
when someone’s private interests may be incompatible or in conflict with that individual's 
public responsibilities. The Ministry is responsible for monitoring potential conflicts of 
interest for its current and former employees as well as (by virtue of agreements with 
program boards) for many government appointees. All boards must notify the responsible 
Minister of any potential conflicts of interest so that corrective action can be taken if 
necessary. 
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The Ministry relies on individuals from both the academic and the industry research 
communities to provide their expertise by reviewing research funding proposals and sitting 
on program boards and panels. Many program advisory boards, which typically comprise 
academic, industry, and government appointees, recommend to the Minister which 
research proposals should be funded. While board and panel members generally receive 
little or no remuneration, they may be involved in research proposals presented for funding 
or be shareholders in companies that are partnered with researchers submitting proposals. 
In addition, appointees may sit on several boards and panels. Consequently, clear, consistent, 
and rigorously enforced conflict-of-interest rules should be in place for all science and 
technology programs. However, the Ministry has no procedures in place for monitoring 
potential conflicts of interest. 


Members of program boards and review panels who have an interest in a research proposal, 
through their connection either to a private-sector partner or to the applicant institution, 
are expected to withdraw from any discussion of that proposal for funding. We reviewed the 
board minutes of several programs and noted instances where board members 
appropriately self-declared numerous instances of potential conflicts and withdrew from 
discussions on related proposals. However, we also noted instances, which we brought to the 
Ministry’s attention, where a conflict of interest should have been declared, but there was no 
indication in the minutes that a conflict had been declared. The Ministry did not have 
sufficient information to determine whether all conflicts had been appropriately declared, 
because the Ministry does not require Board appointees or advisory panel members to 
disclose potential or real conflicts of interest either when they are appointed or if their 
circumstances later change. We also found no evidence that the Minister was notified of all 
potential conflicts of interest as required by Management Board of Cabinet directives. 


Recommendation 


To ensure compliance with the government’s post-service and conflict-of- 
interest requirements and to ensure that its science and technology activities 
are conducted in an open, fair, and transparent manner, the Ministry should: 


develop consistent conflict-of-interest policies that apply to all science and 
technology grant programs; 


develop standardized procedures for adequately monitoring potential 
conflicts of interest; and 


inform the responsible Minister of all conflicts of interest as required. 


Ministry Response 


The Ministry agrees that there should be openness, fairness, and transparency 
in the conduct of the science and technology programs. 


The Ministry will establish a working committee to develop consistent conflict- 
of-interest policies for its programs and will work with central ministries in this 
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regard. The Ministry will use the conflict-of-interest provisions applicable to 
members of advisory boards and expert panels under Management Board 
Directives and Guidelines as a basis for the review. 






The Ministry will take any necessary steps to ensure that the policies are 
followed and, where appropriate, will bring conflict-of-interest matters to the 
attention of the Minister. . 









Project Benefits 


The benefits of science and technology programs include the retention of intellectual 
property rights and industry support. Retaining intellectual property rights for use in 
Ontario benefits the province economically, because successful research can stimulate 
economic activity (such as creating jobs). Industry support also benefits the province 


economically, because most programs are designed to encourage industry participation, and 
such participation generally encourages commercially beneficial research, which in turn 
benefits the whole economy. 


INTELLECTUAL PROPERTY RIGHTS 


Intellectual property rights represent the legal ownership resulting from research and 
academic activities that can result in patents, trademarks, and copyrights. The owner of 
intellectual property has the right to exclude others from using it, and ownership can be 
transferred or sold. One of the Ministry’s major objectives is to support job creation and 
economic growth. Consequently, grant recipients have a responsibility either to use 
intellectual property in Ontario or to use their best efforts to license the intellectual property 
for use in Ontario. 


We reviewed a number of the Ministry’s science and technology programs to determine 
whether research discoveries were benefiting the province and ultimately its taxpayers. We 
found that the Ministry had no general guidelines for safeguarding its interests regarding 
intellectual property rights. For example, ministry contracts under the Premier’s Research 
Excellence Awards program contained no clauses regarding intellectual property rights. 
Ministry requirements under other programs were equally deficient: 


* The Memorandum of Understanding (MOU) between the Ministry and the Ontario 
Research and Development Challenge Fund Board requires that contracts with 
Challenge Fund grant recipients include an intellectual property clause ensuring that 
the applicant institution possess the right to use the intellectual property necessary to 
complete the project and that it will not dispose of those rights without the province’s 
consent. However, we found that more than 80% of the contracts we reviewed did not 
include the required clause. We were informed that funds had been committed to 
projects that were later delayed indefinitely because of disputes over intellectual 


property rights. 
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¢ The MOU also requires that any intellectual property rights arising from ministry- 
funded research be owned by the applicant institution rather than by researchers or 
private-sector partners. The applicant institution is expected to make mutually agreeable 
commercialization arrangements with its business partners and to demonstrate how the 
project's economic benefits extend beyond the business partners to the Ontario 
economy. This clause is intended to ensure that research funded by the province 
benefits the people of Ontario first. But contrary to program policy, 73% of the 
contracts we reviewed stated that intellectual property could belong to either the 
inventor or the recipient institution, and that it was up to these parties to determine who 
would own these rights. Furthermore, the contracts did not obligate intellectual 
property owners to confer the benefits to Ontario. Consequently, the province's 
economic interests may not be sufficiently protected. An expert panel’s 1999 report to 
the Prime Minister's Advisory Council on Science and Technology titled Public 
Investments in University Research: Reaping the Benefits concluded that vesting 
intellectual property ownership with university researchers instead of the university itself 


is one of the single biggest factors accounting for lost commercialization opportunities in 


Canada. 















Recommendation : 
To help meet its overall objectives of supporting job creation and economic 
growth that benefits the people of Ontario, the Ministry should: 


e ensure compliance with program policies on intellectual property rights; 


e review existing policies and develop consistency among programs 
regarding the ownership of intellectual property; and 


e formally assess the various programs’ success in meeting their objectives. 


Ministry Response 





The current trend across North American jurisdictions is to permit research 
institutions to vest the intellectual property rights with either the institution or 
the researcher. Accordingly, the Ministry will establish a working committee to 
work with central ministries and review the policies on intellectual property 
rights for all the science and technology programs to assess the best 
approach to achieve ministry objectives and ensure all research institutes in 
Ontario are treated fairly and have access to funding from the science and 
technology programs. 


INDUSTRY SUPPORT 


Industry participation is required for most ministry programs to help ensure that the 
research being conducted is of commercial interest to the business community and will 
therefore bring about economic growth. The private sector’s interest in the research 
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conducted influences its willingness to support the research through various forms of 
contributions. In-kind contributions include such items as equipment, proprietary software, 
researchers’ salaries, and overhead. Such contributions must be properly valued to ensure 
that each funding partner contributes its fair share as established under the funding 
program policies. 


We reviewed industry’s cash and in-kind contributions to ministry-funded projects and 
found that the Ministry had not ensured recipient compliance with program requirements 
for industry participation or established consistent policies for assessing in-kind 
contributions. For example: 


* The Ontario Research and Development Challenge Fund (Challenge Fund) requires 
that all approved proposals have confirmed private-sector contributions or 
commitments totalling at least a third of the proposal cost “in order to ensure that funds 
are committed to proposals relevant to business and industry.” But 30% of the projects 
we reviewed did not have commitment confirmation letters from private-sector partners 
on file. The unconfirmed commitments totalled $38 million in cash and in-kind 
contributions. Also, the policy requiring a one-third contribution from the private sector 
has not been consistently applied across projects. Ministry contributions ranged from 
4% to almost 60% of the total project cost. While some projects were approved when 
ministry contributions exceeded the one-third guideline, other projects were required to 


find additional funding within a specified period to make up the 33% requirement. 


¢ In-kind contributions are accepted if they are critical to the proposal’s success and if 
their monetary value is assigned in a reasonable manner, normally through third-party 
valuations. The Memorandum of Understanding requires the Challenge Fund Board to 
develop criteria for the valuation of in-kind contributions and to ensure that 
applications conform to these criteria. Fair valuations of private-sector contributions are 
necessary to ensure that the Challenge Fund is not contributing disproportionately to 
projects by matching in-kind contributions whose values have been inflated. But at the 
time of our audit, despite $113.6 million in private-sector in-kind contributions having 
been reported, no criteria for the valuation of in-kind contributions had been put in 
place. We reviewed each private-sector in-kind contribution valued at more than 
$2 million and found no evidence on file of independent third-party evaluations to 
confirm that reported in-kind contributions had been fairly valued. 
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Recommendation 


To better ensure that the required private-sector contributions are actually 
made, the Ministry should: 


e verify that the required commitment confirmation letters are received 
before funding research projects; 


¢ consistently apply the criteria for proportionate program funding and 
document justification for any exceptions; and 
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¢ develop policies for the independent valuation of in-kind contributions. 


Ministry Response 


The Ministry agrees that private-sector contributions must be made for the 
research projects and has placed the responsibility for obtaining the private- 


sector funding with the research institute under the contract agreement with 
the Ministry. 


The Ministry will ensure that the private-sector contributions are made for each 
project, will document exceptions, and will develop appropriate policies for 
independent valuation of all in-kind contributions of a material value. 


PROGRAM FINANCIAL AND ADMINISTRATIVE 
CONTROLS 


Program Administration 


The Science and Technology Division has approximately 50 staff and annual direct 
operating expenses of about $7.8 million. The Division paid out more than $1.3 billion in 
transfer payments between April 1, 1998 and March 31, 2003 and has made 
commitments to spend $3.0 billion more. Advisory boards provide the Minister with a list 
of projects to approve for funding, often without the benefit of a review by and assurance 
from ministry staff that all significant policies have been complied with. We were informed 
that the Minister had approved every grant recipient recommended for funding by the 
advisory boards of the Ontario Research and Development Challenge Fund (Challenge 
Fund), the Premier’s Research Excellence Awards, and the Premier’s Platinum Medal for 
Research Excellence. 


We reviewed the Division’s financial and administrative controls and concluded that 
controls over its funding of transfer-payment recipients needed to be improved. Some of 
our concerns are as follows: 


¢ For the Challenge Fund program, program policy requires that a contract between the 
province and the applicant be signed within 90 days of project approval and that 
funding not be provided until a signed contract is in place. Contractual arrangements 
are necessary to ensure that the interests of the province, the institution, the researcher, 
and the private-sector contributors are protected and that each participant's roles and 
responsibilities are clear. Since the program’s 1997 inception, more than 100 projects 
have been approved, for a total of $435 million in program funding. But almost 30% 
of these projects, approved to receive a total of $128 million from the Challenge Fund, 
were without contracts. Although the program administrator had ensured that no 
money had been paid out for projects with no contracts, $128 million in program 
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funds had been committed to inactive projects where the average length of time since 
the project was approved was more than 18 months and some projects had been 
approved up to three years ago. 


Several science and technology programs provided funds in excess of the recipients’ 
current needs. For example, other ministries prepaid funds to recipients of top-up grants 
from the Ontario Research Performance Fund that resulted in a corresponding 
prepayment by the Ministry. In addition, the Ontario Innovation Trust was provided 
with $510 million more than its current needs, and as of the beginning of the 2002/03 
fiscal year, the Trust had earned $72 million in interest. Conversely, the Ministry made a 
conscious effort to reduce the prepayment of funds to the Ontario Centres of 
Excellence, and as a result, the government may have saved more than $500,000 in 
interest annually. 


A member of the Ontario Research and Development Challenge Fund Advisory Board 
and a member of the board of the IIO (essentially a private corporation) signed a 
contract with an individual to work for the Challenge Fund. This individual was to 
receive $100,000 annually, plus office and travel expenses. However, neither board 
member had the authority to hire staff for a ministry program. 


Although the Ministry acknowledges the need for information systems that provide 
timely, relevant, accurate, and complete program information, no such systems are in 
place to help ministry staff manage transfer-payment programs. Within the Ministry, 
there is a reliance on computer-assisted or manually maintained lists of projects and key 
activities that may need action. A well-designed management information system could 
capture the key data necessary for tracking performance and compare those data against 
contracted milestones and established targets, for the program as a whole and for 
individual grant recipients. 


Recommendation 


To improve the Division’s financial and administrative controls, help achieve 
due regard for economy, and improve staff efficiency, the Ministry should: 


@ 


assess the continued merit of any approved research projects that are 
inactive and, where necessary, terminate funding commitments to inactive 
projects; 

review prepaid funding, so that payments are made to cover only current 
needs; 

clarify the roles and responsibilities of advisory board members and others 
involved in administering science and technology programs; and 

develop an information system to provide the Ministry’s staff with the 
information needed for effectively overseeing its transfer-payment 
programs. 
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Ministry Response 






The Ministry agrees and notes that, over the past two years, the number and 
size of the science and technology programs have increased significantly. As a 
consequence, the Ministry is undertaking a review to determine the level of 
resource requirements required to effectively administer these programs. 


The Ministry will also review all projects that are currently inactive and take 
appropriate action to restart or terminate any projects identified by March 31, 
2004. The Ministry is also currently developing an information system for 
transfer-payment programs. 


The Ministry will assess the need for prepaid funding and clarify the role of 
advisory board members as part of its review of administrative policies. 


The Ministry will clarify the roles and responsibilities of advisory board 
members with respect to the administration of any programs. 


Innovation Institute of Ontario—Administration 


In June 2000, the Innovation Institute of Ontario (IIO), a subsidiary of the Ontario 
Innovation Trust, was created to provide support services for the Trust and potentially for 
other ministry programs. In July 2000, the Ministry requested permission to single-source 
administration of the Ontario Research and Development Challenge Fund (Challenge 
Fund) to the IIO. But contrary to the government’s Framework for Alternative Service 
Delivery, the Ministry did not perform a detailed analysis of alternative delivery options. In 
addition, the Ministry did not want to issue a request for proposals (RFP) because once an 
REP is out, anyone who is eligible must be treated fairly in the process, and problems could 
arise if a bidder was not dealt with fairly after an investment of time and resources in the 
preparation of a proposal. Single-sourcing, especially for this reason, contradicts the basic 
principles of government procurement. 


The Ministry stated that the IIO was designed to provide administrative and support 
services in the most cost-effective way possible. But no cost analysis was done to support 
outsourcing. Furthermore, the Ministry noted that the outsourcing would not lead to 
substantial cost savings, and although a number of qualitative benefits were expected, the 
Ministry did not subsequently assess the outsourcing arrangements to determine whether 
these benefits were realized. 


The Division’s payments to the IIO for administering the Challenge Fund are to be based 
on actual costs incurred and are to be negotiated every year based on operating plans and 
budgets submitted by the IO. But neither negotiations with the I[O nor submissions of 
annual operating plans and budgets by the IO are timely. At the time of our audit in 
March 2003, the Ministry had not received the budget for the fiscal year ended 

March 31, 2003. Likewise, the fees for the 2001/02 fiscal year were not approved until 
after that year ended. 
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Moreover, the Ministry does not know whether the amounts being charged by the IIO are 
reasonable. The Ministry does not receive audited financial statements from the IIO. Such 
statements could provide assurance that expenses reported by the IIO were incurred as 
reported. We requested financial statements from the IIO and received unaudited 
statements for the 2000/01 and 2001/02 fiscal years. Although the expense categories in 
these financial statements differed from those in the funding requests, the salaries and 
benefits paid to the ITO (based on the funding request) for administering the Challenge 
Fund exceeded the salaries reported for the corporation as a whole in the financial 
statements by $200,000 in the 2000/01 fiscal year and by $120,000 in the 2001/02 fiscal 
year. This may be partly due to the difference between the financial statements’ expense 
categories and those in the funding requests. However, the Ministry had not identified or 
followed up on those discrepancies. 


The Challenge Fund should pay only a fraction of IIO costs, because the IIO provides 
administrative services to other ministry-funded programs. But the Ministry has never 
received a workload assessment to justify the amounts being charged to the Challenge Fund 
and other government-funded programs. In addition, in its first two years of operations, the 
I1O, which is a not-for-profit corporation, received total revenues of $4 million, which 


exceeded total expenses by $700,000. 


Furthermore, it is unclear how expenses are being managed between the ITO and its parent, 
the Ontario Innovation Trust. For example, the Trust’s 2001/02 financial statements 
disclosed a $400,000 loan to IIO. According to the agreement between the Ministry and 
the Trustee, Trust disbursements are allowable only for the purposes of funding eligible 


research projects and administering the Trust, not for providing loans to other entities. 





Recommendation 


To ensure that the fees paid to the Innovation Institute of Ontario (IIO) for 
administering the Ontario Research and Development Challenge Fund 
(Challenge Fund) are reasonable, the Ministry should: 


e assess whether the expected benefits of outsourcing have been achieved; 


e insist on receiving a budget and operating plans from the IIO before each 
fiscal year begins, instead of after the year has been completed; 


¢ ensure that it receives audited financial statements from the IIO for use in 
assessing the appropriateness of fees charged for administering the 
Challenge Fund; and 


e ensure that the detailed breakdown of the budget submissions correlates 


with the expense categories used in the financial statements and follow up 
on any discrepancies. 
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Ministry Response 


The Ministry agrees that the fees paid to the Innovation Institute of Ontario 
(lO) for the administration of the Ontario Research and Development 


Challenge Fund (Challenge Fund) should be reasonable and transparent and is 
working with the IIO to improve both the timing of reporting and reports 
received; areas in which reports would be improved would include the details 
for services provided and fees charged. 





The Challenge Fund is on the Ministry’s list of programs to be evaluated for the 
2003/04 fiscal year, per Management Board guidelines. The service agreement 
will be reviewed as part of this program evaluation. 





MEASURING AND REPORTING ON PROGRAM 
EFFECTIVENESS 





“VFM Section 3.07 


Program Planning 


An organization’s mandate identifies its overall purpose, provides general direction, and 
helps in developing specific plans for implementation. The Ministry’s mandate is to foster an 
Ontario with competitive businesses and a prosperous economy by promoting innovation, 
economic growth, and job creation. The Science and Technology Division endeavours to 
contribute to this mandate by building a culture of innovation, investing in people and 
infrastructure to create knowledge, and creating a supportive climate that fosters 
commercialization of research and brings knowledge to the marketplace. But the Division 
does not have a strategic plan with clearly stated goals that will help it accomplish its 
mandate. 


Since the Ministry does not directly manage many of the programs funded by the Science 
and Technology Division, it is essential to have not only appropriate accountability 
mechanisms and clearly stated goals but also inter-organizational collaboration. But as noted 
previously, the Ministry is not provided with enough information on the activities of some of 
its programs to properly co-ordinate funding activities. For example, the Division receives 
virtually no information from the Ontario Innovation Trust regarding its spending of 

$750 million of public funds, and the Division is not apprised of the Trust’s key objectives, 
funding priorities, or future plans. The Ministry is accountable for co-ordinating science 
and technology research throughout the province. Such co-ordination would include 
setting program parameters and developing detailed policies that provide a level of 
consistency and guidance for the Trust, the programs’ advisory committees, and other 
decision-makers. 


Strategic directions encompassing clear goals, priorities, and expected results should guide 
research decisions, program development, and ultimately the selection of specific research 


projects. Since the 1997 inception of the Ontario Research and Development Challenge 
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Fund, the Ministry has announced more than a dozen new science and technology transfer- 
payment programs, some of which target highly specific areas of research. These program 
commitments range from $10 million to more than $1 billion. But the announcement of 
new programs did not result from executing a comprehensive plan or from systematically 
assessing long-term provincial needs. 


The Ministry has reported a strategic framework for Ontario’s science and technology 
programs. This strategy outlines a number of concepts that build towards high-value-added 
jobs in the province. However, the Ministry does not have a detailed plan outlining how it 
intends to reach this objective. The Ministry’s science and technology activities need an 
overall strategic plan to set parameters and consistent policies for existing programs and to 
guide the development of new programs. 





"Recommendation 


To formalize its co-ordination responsibilities and provide clear direction for 
program development and delivery, the Ministry should: 


e review all research programs and prepare a detailed strategic plan that sets 
specific goals and objectives for research in the province; and 

¢ outline policies—such as conflict-of-interest rules, project selection criteria, 
and monitoring guidelines—that all programs must follow regardless of the 
delivery mechanism. 





Ministry Response 


The Ministry agrees there should be clear direction for its research programs 
and will continue to refine and update the strategic framework to ensure its 
relevance for supporting research in Ontario. | 


The Ministry supports clarifying its goals and objectives for supporting 
research and development activity in Ontario. 


In light of the audit findings, the Ministry will review the program delivery 
policies with internal and external experts for all research programs and 
establish a consistent approach to conflict of interest, selection criteria, 
contract agreements, and monitoring guidelines. Any subsequent policies will 
need to take into consideration the differences in objectives for the various 
programs. 





Effectiveness Reporting 


Through the government's business planning process, the Ministrv.is required to report to 
the Management Board of Cabinet and ultimately publish a combined business plan and 
annual report that outlines plans for the coming year and reports on performance from the 
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previous year. [he business planning process is meant to improve decision-making, support 
the development and delivery of high-quality programs, ensure that program outcomes are 
aligned with overall government priorities, and improve the Ministry's accountability for 
measuring and achieving results. Published business plans and annual reports are also 
intended to support openness and accountability to the public and the Legislature. 


We reviewed the Ministry’s business planning process. The Ministry outlined a number of 
performance measures for its core businesses, each of which had targets and commitments. 
Although some actual results were reported internally, the Ministry did not measure and 
report publicly on the achievement of its overall goals of promoting innovation, economic 
growth, and job creation. Only one performance measure was reported in the Ministry's 
published Business Plan for the 2002/03 fiscal year. This performance measure, the 
anticipated growth in the value of non-government-sponsored research over the next five 
years, is future oriented and does not reflect the impact of the Ministry’s efforts on achieving 
its goals. 


In previous years, the Ministry’s goals included growth in Ontario's health and 
biotechnology industry, increased participation in the digital economy, and job growth in 
Ontario’s knowledge-based economy as compared to selected jurisdictions. However, we 
were informed that these goals and their related performance measures were dropped 
because the Ministry’s impact was largely indirect and because the Ministry depended on 
the private and academic sectors to successfully partner with the government. 


To determine whether the Ministry had developed clearly stated performance measures for 
individual programs, we reviewed all science and technology programs with transfer- 
payment commitments of $10 million or more. The major science and technology 
programs published their results annually, although those reports were often out of date. 
Many programs reported only a list of grants, research success stories, or activity-based 
measures, such as the number of projects funded or the money contributed by the private 
sector. Such reporting does not reflect what impact the Ministry’s funding initiatives have 
had on promoting innovation, economic growth, and job creation. In addition, the lack of 
timely reporting precludes the Ministry from taking any necessary corrective action in a 
timely manner. 


Of all the Ministry’s funding initiatives, the Ontario Centres of Excellence had developed 
the most comprehensive performance measurement and reporting system. The centres 
reported that from 1998/99 to 2001/02, as a result of their sponsored research, 98 spin-off 
companies started up, 536 people were employed, 406 patents and registered copyrights 
were issued, and an additional $30 million investment was generated from the private 
sector. Although these are measures of a degree of success, the Ministry has not established 
targets or benchmarks against which to measure such performance results so that success 
and failure can be readily determined. 


In December 2002, the Ministry published the Ontario Innovation Index. This report is 


intended to measure the province’s transition to a knowledge-based economy, showing how 
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well Ontario is performing in comparison with other jurisdictions and identifying areas 
needing improvement. The index reports on 30 indicators of community awareness, 
infrastructure, knowledge transfer and use, and economic and societal outcomes. The index 
should be a useful tool for assessing provincial performance. But the indicators were not 
designed to isolate the impact of the Ministry’s science and technology programs. The 
Ministry has no direct impact on some indicators—such as student enrolment in secondary 
school science courses. The Ministry’s activities do impact indicators relating to innovation 
performance, such as the number of patents, the number of business startups, and 
technology adoption rates. But the Ministry would still have to assess the degree to which its 
activities had impacted on these province-wide measures. 





Recommendation 


To provide better accountability to the public and the Legislature for its use of 
public funds, the Ministry should: 


e develop performance measures, targets, and benchmarks that reflect its 
accomplishments and contributions to the overall goals of promoting 
innovation, economic growth, and job creation; 


¢ perform the necessary assessments to measure whether its initiatives are 
effective in achieving overall ministry goals; and 


e report on the actual achievement of these measures, explaining any 
significant deviations from established targets and benchmarks. 


Ministry Response 


The Ministry is committed to measuring the contribution of its programs in. 
promoting innovation, economic growth, and job creation in Ontario and will 
identify and adopt a funding focus to science and technology research and 
development programs that is consistent with current global trends and best- 
in-class approaches to supporting innovation. 





The Ministry will establish feasible performance measures to demonstrate how 
the science and technology programs contribute to ministry objectives. It will 
assess and report on achievements and will refine and i une these 
measures as required. 

y. pe ei 








196 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





MINISTRY OF THE ENVIRONMENT 


3.08—Environet 





BACKGROUND 


The Ministry of the Environment has a broad mandate to restore, protect, and enhance 
Ontario’s environment to ensure public health and environmental and economic vitality. It 
works to ensure cleaner air, water, and land through many Acts and associated regulations, 
which include the Ontario Water Resources Act, the Environmental Assessment Act, and the 
Environmental Protection Act. 


In 2000, the Ministry developed a new information technology vision and strategy called 
Environet to strengthen the delivery of its environmental programs. The Environet envisages 
and provides for individual management information systems to collect and report data that 
ministry management and staff need to enhance their inspection and other compliance 
work. The systems being developed as part of the Environet strategy include: 


¢ a Drinking Water Information System (DWIS) incorporating electronic submission of 
water quality test results from laboratories that analyze water samples from Ontario's 
waterworks facilities; 


¢ a Hazardous Waste Information Network (HWIN) to replace the legacy system, 
Hazardous Waste Information System (HWIS), for tracking hazardous waste 
movements throughout the province; 


¢ an Air Emission Registry (OnAir) that receives emission reports electronically from 
facilities emitting airborne contaminants and posts this information on a public Web 
site; and 


¢ a Computer Assisted Mobile Enforcement Office (CAMEO) system that provides a 
SWAT inspection group with the ability to access and maintain inspection information 


while in the field. 


At the time of our audit the Ministry had spent approximately $17.1 million developing 
these applications. 


The Environet is a long-term strategy and as such is being designed to ensure flexibility in 
response to future legislative and regulatory changes. Several of its planned components 
have yet to be finalized. Through the use of the Environet systems—along with strong 
deterrents and aggressive inspection and enforcement activities—the Ministry plans to 
strengthen its environmental analysis and reporting capabilities, improve industry 
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compliance with its regulations, and fulfill the government's future environmental 
commitments. 


AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of the Environet were to assess whether the Ministry had 
adequate policies and procedures in place to: 


¢ ensure that Environet systems adequately addressed existing legislative and regulatory 
requirements; and 


* ensure that the Environet systems are being developed in accordance with government 
and IT best practices and the government’s environmental commitments for which the 
Ministry is responsible. 


At the beginning of our audit, we identified audit criteria that would be used to address our 
audit objectives. These were reviewed and accepted by senior ministry management. 


Our audit was substantially completed by the end of March 2003. The scope of our audit 
included discussions with ministry staff and a review and analysis of relevant policies, 
procedures, and related documents on the four major Environet applications listed earlier. 


As part of our audit, we used a number of Computer Assisted Audit Techniques (CAATs) to 
extract and analyze ministry and industry data housed in the databases supporting each 
system. Given the importance of the Ministry’s inspection activities in ensuring compliance 
with environmental regulations, we reviewed the use of the Environet systems by inspectors 
and followed up on a number of inspection issues made in prior audit reports. 


We did not rely on the Ministry’s internal auditors to reduce the extent of our work because 
they had not conducted any recent work in the areas covered by the audit. 


Our audit was performed in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants. Accordingly, it included such tests and other procedures that we 
considered necessary in the circumstances. 


OVERALL AUDIT CONCLUSIONS 


We concluded that, at the time of our audit, the Ministry’s Environet systems did not 
provide ministry staff with the information needed to support the Ministry’s responsibilities 
of ensuring that drinking water meets regulatory standards, that hazardous waste 
movements are properly controlled, and that all air emissions are monitored and reported 
where required. We further noted that the Environet systems do rot, as yet, enable the 
Ministry to summarize and analyze regulatory compliance and to maximize inspection 
efforts by identifying areas of highest risk. The Ministry continues to rely on legacy systems 
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and manual procedures in some instances. More effort will also be needed to ensure 


Environet systems adequately support all the Ontario government's environmental 


commitments. Our major findings were: 


The Drinking Water Information System (DWIS) did not identify whether all 
waterworks were submitting their required sample test results. We noted that 300 of the 
1,476 currently registered non-municipal waterworks have never submitted any test 
results to the Ministry. The Ministry is still following up on 237 of these facilities. 
Furthermore, 612 or approximately 27% of registered waterworks had not submitted 
the minimum number of water samples for two of the highest-risk substances, E. coli 
bacteria and fecal coliform. 


Many waterworks continue to experience compliance difficulties including exceedances, 
which are water samples with unacceptably high concentration levels of regulated 
substances. Although exceedances represent less than 1% of total samples taken, our 
data-extraction work did identify 6,725 exceedances since 2001. Of these, 3,181 were 
identified as Adverse Water Quality Incidents (AWQI), which are more serious 
exceedances that affect human health. 


The ministry systems and procedures did not ensure all AWQI were reported and 
followed up on. For example, the Ministry was not aware of 31 out of 46 AWQI at one 
waterworks. Moreover, summary reports did not distinguish the more serious AWQI 
from other exceedances, or raw-water samples from treated-water samples. Such reports 
were therefore of little value in ensuring that the highest-risk cases were prioritized for 
investigation. Furthermore, the system did not report on whether problem incidents 
were resolved. 


A new tracking system to accept electronic transaction submissions from hazardous 
waste generators, carriers, and receivers was being used for less than 1% of hazardous 
waste movements. The older, inefficient, paper-manifest system was still being used to 
handle almost all transactions because the new system could not accept paper 
submissions. The new system also had few analysis and reporting capabilities. Although 
we noted that over 5,000 unauthorized hazardous waste movements (which represent 
less than 3% of total movements) were flagged by the system, we found no evidence of 
any follow-up action to investigate and resolve them. 


The new CAMEO application was working well. For example, its mobile capability 
allowed the SWAT team to access ministry data at inspection sites, thus increasing 


efficiency. 


Although the Ministry has recently increased the number of inspectors, total inspection 
activity is currently at 73% of 1995/96 levels and inspectors are averaging fewer 
inspections annually. We were informed that this was due to the increased length of time 
it now takes to conduct inspections. Given the significant increase in the number of 
facilities that are now covered by new regulations, the Ministry needs to develop a 
strategy to deal with these new regulated facilities. For example, last year water inspectors 
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visited only 54 of the 357 private, drinking water treatment plants and 44 of the 1,119 
smaller plants and designated facilities. 





Overall Ministry Response 


The Provincial Auditor’s Report is consistent with ongoing ministry initiatives 
to increase the effectiveness of our Environet systems. The Ministry recognizes 
that its ability to engage in effective inspection, investigation, and enforcement 
activities in all areas of its responsibility—air, water, land—can be enhanced by 
information technology. Over time, the Ministry has developed multiple 
systems, usually on a program-by-program basis, to assist staff by providing 
more effective and efficient data collection, analysis, and reporting. 


Recognizing that an integrated strategy for managing information could prove 
even more useful, the Ministry adopted the Environet information technology 
strategy in 2000. The Environet strategy was designed to provide the Ministry 
with more effective day-to-day information with which to manage current 
programs. 


Since Environet is a long-term strategy, the Ministry is moving forward on a 
timetable that allows for adequate development and testing of new systems 
and a smooth transition from old systems to new. 





DETAILED AUDIT OBSERVATIONS 
DRINKING WATER 


There are more than 1,100 large waterworks facilities in the province that provide drinking 
water for Ontario residents. Municipalities own and operate over 700 of these facilities, 
which service approximately 80% of Ontario households. In addition to these large 
facilities, approximately 1,100 smaller private waterworks provide drinking water to many 
rural areas and to a number of institutions, including schools, day nurseries, social and 


health care facilities, and trailer parks. 


As a result of the May 2000 tragedy in Walkerton that claimed seven lives and caused more 
than 2,300 people to become ill, the Ontario government in August 2000 launched 
Operation Clean Water—an action plan designed to ensure that Ontario residents have the 
safest drinking water in Canada. The cornerstone of this plan was a new Drinking Water 
Protection Regulation under the Ontario Water Resources Act. Covering all municipal and 
large waterworks systems, the regulation set out revised minimum sampling, analysis, and 
reporting standards for each facility. As well, the Ministry adopted a new policy of 
inspecting all municipal waterworks annually. In its first report on municipal facility 
inspections under the new program, inspectors noted that of the 558 facilities visited, 275 
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or 49% failed to conform to established standards and 24% were repeat offenders. Iwo 
hundred correction orders were issued. 


In December 2001, a second regulation, the Drinking Water Protection Regulation for 
Smaller Waterworks Serving Designated Facilities came into effect. This regulation 
significantly expanded the scope of the Ministry’s monitoring responsibilities by introducing 
new sampling and testing requirements for many previously unregulated smaller waterworks 
facilities. 

Over the past year, the Ministry has continued to address drinking water issues. In the two- 
part report of the Walkerton Inquiry released in January and May 2002, Mr. Justice Dennis 
O’Connor made many recommendations for improving Ontario’s water system from 
“source to tap.” The government has accepted these recommendations and has committed 
to implementing all of them eventually. Many were addressed in Ontario’s new Safe 
Drinking Water Act, passed in December 2002. Subsequent to our audit, in May 2003 new 
regulations were enacted to support this legislation. The regulations increase the number of 
waterworks covered by the Act and set out specific sampling and testing requirements for 
each waterworks type. Future initiatives will be necessary to address many of the remaining 
recommendations of the Walkerton Report, in particular, those calling for protection of 
Ontario’s water at source (that is, prior to being processed by waterworks facilities). 


Drinking Water Information System (DWIS) 


Since 2000, the Ministry has been developing a Drinking Water Information System 
(DWIS) to support its new monitoring and reporting requirements. DWIS was designed to 
allow the electronic submission of water quality test results to improve timeliness and reduce 
re-inputting errors. It also partially addressed a number of Justice O'Connor's 
recommendations calling for improvements to the Ministry's information systems. In 2001, 
the Ministry launched an interim Web-based application to register all waterworks and 
receive water-sample test results electronically from laboratories servicing them. This interim 
application was to be replaced in March 2002 by the full version of the new DWIS, an 
Environet application that would provide additional features. For simplicity, we refer to 
both the interim and the planned system as DWIS. The Ministry had spent approximately 
$6.9 million to develop this system at the time of our audit. 


By October 2002, the Ministry did implement significant components of the new system. 
DWIS can now register large waterworks, accept their quarterly reports, and record their 
Adverse Water Quality Incidents (AWQI). At the time of our audit in March 2003, 
however, key components of DWIS remained to be implemented. The Ministry had not 
completed the registration of all smaller waterworks and was still working to complete the 
fully integrated DWIS application. Perhaps more importantly, reporting tools to help 
management track and summarize test submissions for regulatory non-compliance were still 
under development. 
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The DWIS database, which contains all registered waterworks and their reported water- 
sample test results, has great potential to help ministry inspectors plan and prioritize their 
work of ensuring that waterworks facilities are operating properly. Summary reports could 
pinpoint which waterworks present the greatest health risks for Ontarians. Examples of 
reports that could be generated are: waterworks that are submitting significant numbers of 
exceedances or not submitting test results; facilities that are not meeting the frequency 
requirements established by regulation or not submitting reports on time; and laboratories 
that do not follow proper testing procedures, that are not accredited for the tests they are 
performing, or that are not properly communicating AWQI to all required parties. 


DWIS is also used to record basic profile information on all registered waterworks; however, 
we found some of the database records to be inaccurate. For example, we found 50 
waterworks registered incorrectly in the system and several with inaccurate profile 
information. 


Recommendation 





To ensure that the quality of Ontario’s drinking water is properly monitored 
and that appropriate inspection and other follow-up action is taken on a timely 
basis when necessary, the Ministry should: 


° complete the development of the Drinking Water Information System 
(DWIS) as soon as possible; 


e explore ways to use DWIS and its data to generate reports that would help 
inspectors identify and prioritize candidates for inspection and summarize 
waterworks regulatory compliance; and 

e improve validation procedures to ensure all waterworks records in DWIS 
are accurate. 


Ministry Response 





The Ministry acknowledges the importance of completing the development of 
the Drinking Water Information System (DWIS). DWIS development has 
progressed considerably since the audit. 


The Ministry is committed to ensuring that the transition to DWIS is done 
effectively and efficiently, while protecting public health. The Ministry is using 
existing systems to discharge its drinking water responsibilities until DWIS is 
fully operational. Recent investments in both human and financial resources 
will ensure the full operation of manual systems while DWIS is being 
completed. 








The Ministry acknowledges the importance of using DWIS to identify and 
prioritize candidates for inspection. The Ministry has made a strong 
commitment to inspection and enforcement as a key part of its approach to 
manage drinking water. The Ministry annually inspects 100% of all municipal 
water systems that serve over 80% of the population of Ontario. In addition, the 
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Ministry conducts one unannounced inspection of municipal water systems for 
every three proactive inspections. 


The Ministry will continue to explore ways to use DWIS as part of its overall 
risk assessment processes to help inspectors identify and prioritize 
candidates for inspection and summarize waterworks regulatory compliance. 


The Ministry ensures the waterworks records held in DWIS are accurate by 
conducting Quality Assurance/Quality Control on all waterworks profile 
information received from waterworks by reviewing data against other ministry 
databases and contacting waterworks to resolve any discrepancies or obtain 
missing data. 


Once fully operational, DWIS will allow for further improvements to data 
accuracy and cross-checks in the maintenance of waterworks profiles. The 
Ministry will require waterworks to use “smart forms” to submit profile 
information—these forms have a built-in capacity to conduct primary 
validation of data. Ministry staff will continue to review all data and conduct 
secondary validation as necessary (for example, by contacting waterworks to 
resolve discrepancies). 








Submission of Water-Sample Test Results 


All waterworks are required to submit water samples at regular intervals to an accredited 
laboratory to test for the presence of many chemical or organic substances. The laboratory 
then submits the test results electronically to the Ministry via DWIS. We found that DWIS 
was unable to identify whether all waterworks were submitting these required sample test 
results. Accordingly, we used data-extraction procedures and our results indicated that while 
municipal waterworks generally did submit samples, the Ministry had never obtained test 
results from 300 of the 1,476 registered non-municipal waterworks. The Ministry followed 
up on and resolved 63 of these 300 waterworks; however, the remaining 237 were still 
being reviewed at the conclusion of our audit fieldwork. 


Furthermore, the regulations of the Ontario Water Resources Act specify a minimum 
frequency of testing on certain high-risk substances, such as Escherichia coli (E. coli) bacteria 
and fecal coliform. We performed additional data-extraction work to summarize all test 
results submitted since DWIS became operational and found that for 612 (approximately 
27%) of the registered waterworks, the Ministry did not obtain the minimum number of 
water samples for these two substances. 










Recommendation 


To enhance its ability to respond to water problems promptly, the Ministry 
should improve controls to ensure all waterworks submit their water-sample 
test results and compliance reports in accordance with regulatory 
requirements. 
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Ministry Response 


The recommendation is fully consistent with the Ministry’s compliance 
objectives. One hundred percent of quarterly test-result reports from municipal 
water systems that serve over 80% of the population are tracked and followed 
up on. In terms of recent improvements, Drinking Water Information System 
reporting to identify waterworks and laboratories that do not supply required 
information to the Ministry has now been completed. These compliance 
reports, which were not available at the time of the audit, have significantly 
increased the Ministry’s oversight powers by automatically notifying ministry 
inspectors of waterworks and laboratories that are not submitting sample test 
results and reports as required by regulations. 








EXCEEDANCES AND ADVERSE WATER QUALITY INCIDENTS 


A water sample is considered an exceedance whenever it contains more than the maximum 
acceptable concentration limit for substances as specified by the Ontario Water Resources Act 
regulations. [reated-water exceedances are further classified as Adverse Water Quality 
Incidents (AWQI) if they relate to potentially serious health concerns. Ministry policy is to 
follow up on all AWQI to determine their cause, to ensure appropriate corrective action is 
taken, and to notifiy the public where necessary (such as an advisory to boil water). 


Given the increased number of facilities now requiring monitoring, the past compliance 
problems with many of them, and the available inspection resources of the Ministry (see 
“Inspections and Management Information Systems” later in the report for more 
information), a good system to plan and prioritize follow-up activity is critical. While we 
noted significant numbers of reported water quality problems flagged in DWIS, the system 
did not provide sufficient information to help ministry staff deal with these problems 
effectively. Specifically: 


¢ DWIS reports did not distinguish AWQI from other exceedances. AWQI are 
considered the most significant type of exceedance and demand immediate investigation 
and resolution because of the potential health risks associated with the substances in 


question. Therefore, the system should highlight AWQI exceedances. 


e Waterworks and laboratories submit both raw-water (untreated or source) and treated- 
water (processed and ready for distribution) test samples into DWIS. Raw-water 
exceedances may not be indicative of quality problems if treatment removes unwanted 
substances. At the time of our audit, DWIS reports did not distinguish between raw- 
and treated-water exceedances separately. Also, the system did not have sufficient edit 
controls to reject obvious conflicting test results, such as where samples were reported as 
“distributed” or “treated” even though they had been taken from an “untreated” sample 
location. 
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¢ In our sample testing of the maximum acceptable concentration limits maintained in 
DWIS, we found that the system had not always been updated when new or amended 
regulatory standards came into effect. For example, the acceptable concentration limits 
for uranium and bromate were amended in July 2002, but this information had not 
been uploaded onto DWIS. If updates are not kept current, DWIS cannot accurately 
flag results where acceptable concentration levels have been exceeded. 


Once we brought a number of suspect AWQI] issues to its attention, the Ministry quickly 
investigated and found that most were raw-water samples rather than treated-water samples 
(that is, water for drinking). Others were older AWQI that had not been properly reported 
by a few laboratories. However, our concern is that future incidents could occur and not be 
properly investigated without improved DWIS reporting capabilities. 


Because of the difficulties with existing DWIS reports, we used computer-assisted 
techniques to extract only treated-water exceedances from the database. As the following 
table summarizes, many waterworks continue to report exceedances including the more 
serious AWQI. Although relatively infrequent (less than 1%) in comparison to the total 
samples in the DWIS database, our data extraction identified 6,725 exceedances over the 
past two years. Of these, 3,181 were classified as AWQI. 


Analysis of Water-Sample Test Results 
Submitted to the Ministry 


: ‘Test Waterworks | 
_ TimePeriod | Results with — Exceedances | AWQI' 
Received | Exceedances _ 


2001 555,544 376 SAG wilet oat 


2002 1,230,474 850 3,878 Lom 


| Use Se 
|| 1,901,227 | 





1 
The AWQI figures include incidents relating to E. coli/fecal coliform, total coliform, 
heterotrophic plate count (HPC), background coliform, and sodium. 


2 
No total is provided to avoid double counting waterworks that had exceedances in 
more than one year. 


Source of data: Ministry of the Environment 


Recommendation 


To improve its ability to investigate and resolve water problems promptly, the 
Ministry should: 


e enhance the existing system to highlight all Adverse Water Quality 
Incidents for management attention to ensure timely follow-up action; and 
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e promptly update substance concentration limits to reflect new and 
amended standards. 


Ministry Response 


The Ministry takes all Adverse Water Quality Incidents (AWQI) very seriously 
and has systems in place to respond. The Ministry uses a science-based 
framework for classifying AWQI as high or low risk. Where a high-risk problem 
has been identified, the Ministry responds with immediate (24/7) on-site 
inspectors who undertake the appropriate response. To better ensure all high- 
risk AWQI are responded to as soon as possible, the Ministry will develop 
Environet reports that highlight AWQI. In the case of low-risk AWQI, the 
Ministry has established protocols for staff to use in determining the 
appropriate response. 








The recommendation to update substance concentration limits promptly is 
consistent with the Ministry’s approach. The Ministry has put procedures in 
place to ensure that new standards, once brought into effect through a new or 
amended regulation, will be input into the Drinking Water Information System 
on a timely basis. 





REPORTING OF ADVERSE WATER QUALITY INCIDENTS 


The Ontario Water Resources Act regulations require laboratories and the waterworks owners 
to report any AWQI immediately to the local medical officer of health and the Ministry. 
These notifications, which must be by telephone to live persons, are in addition to the 
requirement to submit all sample results electronically into DWIS. Upon receiving such 
notification, ministry staff must investigate the incident to ensure that all AWQI are 
properly resolved. 


Our data-extraction testing identified several waterworks with significant numbers of 
AWQI. We investigated what follow-up actions had been taken on these incidents and 
concluded that the Ministry needed to improve its AWQI monitoring and tracking 


procedures: 


* Our testing indicated that not all laboratories were notifying the Ministry as required 
when AWQI occur. For example, for one of our sample waterworks, the Ministry was 
not aware of 31 out of the 46 AWQI that were identified in DWIS, since the system did 


not generate an exception report from the electronically submitted data. 


* Conversely, we found instances where laboratories had reported AWQI to the Ministry 
in accordance with the verbal notification requirements but did not submit the sample 
results electronically into DWIS. We also noted instances where AWQI were attributed 
to the wrong waterworks. A reconciliation of the verbal AWQ\) notifications and the 
submitted water-sample results would help ensure that all AWQI have been properly 


investigated. 
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e After laboratories report AWQI to the Ministry, DWIS is used to track proper 
notification of the incident to all required parties. However, DWIS is not used to record 
follow-up action, corrective orders issued, or incident resolution. Inspectors document 
these efforts on a province-wide system that is not currently linked to the suite of 
Environet applications. Since these functions are not integrated, the Environet system 
cannot produce reports that allow management to easily monitor whether all AWQI are 


being addressed. 










Recommendation 


To ensure that all serious water problems are corrected, the Ministry should 
consider incorporating a follow-up reporting/resolution module within the 
Drinking Water Information System that would provide information to 
management about incident resolution for each Adverse Water Quality 
Incident. 


Ministry Response 













Information about incident resolution for each Adverse Water Quality Incident 
(AW@Q)) is currently contained in a province-wide system. When an AWQI 
occurs, the Spills Action Centre creates an incident report. The information will 
then be sent automatically to the proper district office for follow-up activities, 
and the resolution for the incident is recorded. The new Drinking Water 
Systems Regulations effective May 2003 make it mandatory that a report be 
submitted to the Ministry for every AWQI no lIater than seven days after the 
issue has been resolved. The Ministry uses both Environet and non-Environet 
systems in a comprehensive management information system. The Ministry is 
developing a strategy that will assist in meeting the longer-term solution of full 
integration of both Environet and non-Environet systems. 


HAZARDOUS WASTE 


More than 2 million tonnes of hazardous waste are generated in Ontario every year. Much 
of this waste must be transported to approved treatment facilities located throughout the 
province. Transporting waste entails the risk of spills and seepage that could cause 
considerable environmental damage. We noted that in its recent inspection sweep of 
hazardous waste generating and processing facilities, the Ministry found compliance 
problems with all 22 facilities inspected. For instance, unapproved handling or treatment of 
waste occurred at 95% of these facilities. Additionally, inspectors found six hazardous-waste 
generators that had failed to register with the Ministry. 
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Hazardous Waste Information Network (HWIN) 


Ministry regulations require tracking documents called manifests to be used to record all 
hazardous waste movement in the province. Each waste generator, carrier, and receiver 
completes separate parts of a manifest, and both the generator and the receiver must submit 
manifest copies to the Ministry. There are approximately 225,000 hazardous waste 
movements annually. 


In late 2001, the Ministry began developing an Environet application known as the 
Hazardous Waste Information Network (HWIN). HWIN would replace the previous 
Hazardous Waste Information System (HWIS) and was designed to eliminate paperwork 
and reduce processing errors by enabling the electronic submission of both manifests and 
fees. The HWIN system was implemented in February 2002 and the Ministry had spent 
$4.1 million on this system at the time of our audit. 


Since a key goal of HWIN was to move to an electronic-manifest system, it was developed 
without the capability to support paper manifests. However, this lack of functionality has 
proven problematic. All three parties to a hazardous waste movement transaction—the 
generator, the carrier, and the receiver—must be equipped to handle electronic manifests 
before the HWIN system can be used. The result has been minimal use of the new system. 


In its first year of operation, only 1,885 (less than 1%) hazardous waste movement manifests 
were processed through HWIN. The rest were still handled by the paper-manifest system. 
Accordingly, few of the efficiency goals for the HWIN system have been achieved and both 
systems must be maintained to handle all manifests properly. At the time of our audit, the 
Ministry was addressing this problem through a project to develop a paper-manifest 
component within HWIN. 


As well, at the time of our audit, the HWIN system had few analysis and reporting 
capabilities. For example, it could not produce summary reports of the generation and 
movement of hazardous waste, even though such benefits were cited in the Ministry’s 
business case for the system. Furthermore, it did not generate reports that would highlight 
possible inspection candidates. In fact, we found the former HWIS to have a superior 
reporting capability. Unlike the present system, HWIS could produce reports that showed 
the disposal history of a specific generator or the total amount of waste moved in a given 
timeframe for a specific generator or receiver. 








Recommendation 


To ensure that all hazardous waste movements are properly monitored to 
minimize the risk to the public, the Ministry should: 


e develop and deliver an ongoing incentive, conversion, and communication 
strategy to promote the adoption of electronic manifests by the hazardous 
waste industry; and 
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e develop Hazardous Waste Information Network analytical and reporting 
tools that provide summary information related to the generation and 
movement of hazardous waste and help identify potential problems 
warranting follow-up. 


Ministry Response 


The recommendation is consistent with the Ministry’s current approach. The 
Ministry has initiated a “Hazardous Waste Information Network (HWIN) 
Outreach” strategy to increase and enhance use by clients, thereby improving 
the quality and quantity of information on HWIN. This initiative will provide 
information on the electronic registration, manifest, and fees systems. It will 
provide an open forum in which HWIN users can identify problems with the 


systems and obtain responses from the Ministry. As well, it will allow users and 


the Ministry to identify and develop mutually beneficial options to add value to 
the HWIN system. The Ministry is also developing a supplemental training 
package about the HWIN systems for users that is based in part on feedback 
from hazardous waste generators, carriers, and receivers. 


When resources become available, the Ministry intends to build into HWIN 
analytical and reporting tools that will, for example, provide summary 
information on the generation and movement of hazardous waste. Until HWIN 


has this capacity, the Ministry will continue to obtain such information from the 


present Hazardous Waste Information System. 





REGISTRATION OF HAZARDOUS WASTE FACILITIES 


Every year, the Ministry requires all generators to register in HWIN by submitting details of 


their facilities and the composition of the waste that they are generating. All carriers and 


receivers are also requested to register voluntarily with the Ministry. Registration ensures that 


the Ministry has a complete inventory of all hazardous waste facilities. 


All generators must use ministry-approved carriers and receivers for their hazardous waste. 


Certificates of approval are issued for this purpose and the ministry staff use HWIN to track 


the approved waste types that each carrier and receiver can handle. 


The due date for annual re-registration is February 15. However, we noted that the majority 


of generators fail to register on time, and the Ministry makes little effort to follow up on 
delinquent registrants. For example, although approximately 22,000 generators eventually 
registered with the Ministry in 2002, only 7,986 had registered by the February 15 due 
date. The problem continued in 2003, with only 9,368 generators registered by the 
February 15 due date. Without complete industry information, some facilities may escape 
inspection and therefore compromise the Ministry’s efforts to protect the ecosystem and 


human health. 


Although relatively infrequent in comparison to the total number of hazardous waste 
movements, the HWIN system had flagged a number of unauthorized waste movements 
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since its introduction last year. However, we found no evidence of follow-up action for any 
of these movements. For example, we noted 1,697 (less than 1%) hazardous waste 
movements by unregistered generators, and 3,720 (less than 2%) hazardous waste 
movements for which the generator did not have ministry approval for the type of 
waste moved. Carriers and receivers were also moving waste that they were not 
authorized to handle. 


Recommendation 


To ensure that all hazardous waste is moved in accordance with regulatory 
standards, the Ministry should: 


e ensure all active hazardous waste generators are registered; 


e investigate hazardous waste movements initiated by unregistered 
generators; and 


e investigate hazardous waste movements where the generator, carrier, or 
receiver is not authorized to handle the waste type. 


Ministry Response 


Under current regulation, the requirement to register is the sole responsibility 
of the generator. To reinforce this requirement, the Ministry has provided up to 
three reminders to hazardous waste generators known to the Ministry. 


With respect to investigating hazardous waste movements by unregistered 
generators, the recommendation is consistent with the Ministry’s current 
approach. The Ministry has now developed a comprehensive and integrated 
program for monitoring hazardous and liquid industrial wastes, from their 
point of generation through to their ultimate disposal, of which the Hazardous 
Waste Information Network is one component. 


In order to deal with specific compliance issues related to the hazardous waste 
industry, the Ministry’s Environmental SWAT (originally Soil, Water and Air 
Team) has implemented a targeted compliance strategy over the past 18 
months. During that time, SWAT conducted widespread inspection sweeps of 
hazardous waste transfer and processing facilities, which included examining 
both generator registration and receiver documentation. In addition, SWAT has 
conducted numerous inspections of hazardous waste carriers to, among other 
objectives, verify hazardous waste manifests. 





AIR 


Air pollution is a serious threat to our health, environment, and eccnomy; in Ontario, over 
10 million tonnes of airborne contaminants are emitted annually. The Ministry classifies air 
issues as global, regional, or local. Global air issues include dealing with emissions causing 
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climate change that increases average global temperatures or results in stratospheric ozone 
depletion. Regional air issues include acid rain from the release of sulphur dioxide and 
nitrogen oxides, or smog from the emission of particulate matter, sulphur dioxide, nitrogen 
oxides, or volatile organic compounds. Local air issues include air toxicity from elevated local 
concentrations of compounds, such as metals, volatile organic compounds, soiling, or 
odours. 


In May 2000 and May 2001, respectively, the government introduced new regulations 
under the Environmental Protection Act to increase public accountability for all air pollution 
sources and to improve monitoring and disclosure of environmental pollutants. The first 
regulation, Electricity Generation—Monitoring and Reporting, covered the electricity 
sector, and the second regulation, Airborne Contaminant Discharge Monitoring and 
Reporting, widened the requirements to cover many other industrial sectors. The second 
regulation (O. Reg.127/01) requires all industrial (including the electricity sector), 
commercial, institutional, and municipal facilities subject to the regulation to monitor and 
report on their emissions annually for any of the more than 350 airborne contaminants, 
should their respective annual thresholds be exceeded. Smog season (May | to 

September 30) reporting is required for those criteria air contaminants if their emissions 
exceed their respective annual thresholds. For facilities that maintain large discharge units, 
quarterly reports must also be submitted. 


Ontario Air Emission Registry (OnAir) 


OnAIR is the Environet system designed to provide the public with timely access to 
information on air emissions data reported by the facilities under O. Reg.127/01. These 
data support the Ministry’s air quality initiatives and development of future environmental 
programs. Emissions data in OnAIR replaced those collected under the previous voluntary 
survey reporting program that had a low response rate. OnAIR allows facilities to submit 
emission reports electronically to the Web site. The facilities are responsible for the accuracy 
of their reported emission data. The Web site has been operating since May 2002. The 
Ministry had spent approximately $3.3 million to develop the system at the time of our 


audit. 


We provided our observations and made recommendations on the Air Emission Registry 
(OnAir) to the Ministry and have received satisfactory management responses to our 
recommendations, which included ministry action plans to address them. Our key 
recommendations were that the Ministry should: 


* complete the inventory of the facilities that should be reporting air emissions; 
verify if facilities are approved to emit the substances they report on; and 
° periodically verify the accuracy of data submitted. 


We will follow up on the Ministry's action plans when sufficient time for implementation of 
our recommendations has elapsed. 





Environet AaB 


VFM Section 3.08 


VFM Section 3.08 


INSPECTIONS AND MANAGEMENT INFORMATION 
SYSTEMS 


Physical inspections are critical in assessing compliance with legislative requirements, and 
good management information systems can help inspectors prioritize their activities by 
identifying areas of high risk. In the 2002/03 fiscal year, the Ministry had a staff of 261 
environmental officers—30 were assigned to the SWAT inspection group and 231 worked 
out of five regional offices. Under the Ministry’s new policy, the regional staff are to inspect 
all municipal waterworks facilities annually. As well, they inspect many other areas of 
environmental importance to Ontarians, including air facilities, refrigeration/ozone- 
depleting sites, pesticides, industrial biosolids, municipal biosolids, septage hauler/disposal 
facilities, municipal sewage treatment plants, sub-surface sewage disposal systems, PCB sites, 
and both open and closed waste disposal sites. The Ministry is also developing an inspection 
program for laboratories that conduct water-sample tests. 


During our office visits to different areas, we interviewed a number of regional inspectors 
who conduct inspections of water, hazardous waste, and air facilities. None used the 
Environet applications or their data. Notwithstanding, when we discussed the type of data 
available from Environet, the inspectors commented that this data could be useful in 
identifying appropriate inspection targets. 


We have made many comments about the Ministry’s inspection efforts in the past, dating 
back to the 1980s. For example, in our 1994 audit of water- and sewage-treatment facilities, 
we noted that the drinking water facilities with more significant compliance problems were 
mainly the smaller ones and recommended increased surveillance. In our 2000 audit of the 
Operations Division, we recommended that the Ministry explore options to increase its 
inspection coverage. Since then, the Ministry has introduced the SWAT team to focus on 
priority areas across the province and in January 2001 it hired another 25 water inspectors 
to increase coverage of this critical sector. 


The SWAT group is a province-wide team that uses a risk-assessment framework to target 
specific sectors based on past instances of non-compliance and the potential for major effects 
to human health or the environment. Since its inception in September 2000, the SWAT 
team has conducted more than 2,100 inspections in sectors such as electro/metal-plating, 
hazardous waste transfer and processing, industrial waste haulers, pesticide applicators, and 
auto-body and repair shops. 


The SWAT group makes use of an Environet application—the Computer Assisted Mobile 
Enforcement Office (CAMEO) system—that was introduced in May 2002 to provide 
them with the ability to access and maintain inspection information while in the field. In 
addition to this mobile capability, CAMEO integrates several features that include a case 
management module, time tracking, data analysis, and reporting. The Ministry had spent 
$2.8 million to develop the system at the time of our audit. We reviewed the CAMEO 


application and found that it was working well. 
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The result of the preceding initiatives is an increase of 13%, from 4,182 total ministry 
inspections in 2000 to 4,734 in 2002. As well, in November 2002 the Ministry developed 
a new Municipal Drinking Water Inspections Protocol to address many of the Walkerton 
Report recommendations. 


There are still significant concerns with inspection coverage, however. For example, the 
following chart, which excludes vehicle emission inspections, shows that the new policy of 
annually inspecting all municipal waterworks has had a negative impact on the Ministry's 
abilities to cover other environmental sectors. As well, even after accounting for the new 
SWAT group, inspection activity for 2002/03 is currently at 73% of 1995/96 levels. The 
Ministry has informed us that this can be attributed to the fact that the inspections are now 
more compliance-related and more comprehensive, and therefore they take longer to carry 
out than in the mid-nineties. Notwithstanding, this decrease in the number of inspections is 
worrisome because the number of facilities now covered by the new regulations has 
increased and will continue to do so. 


7,000 5 
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Although inspection coverage of smaller waterworks facilities is currently minimal, there has 
recently been a renewed focus on these facilities. Registration and the submission of sample 
tests are now required from many previously unregulated, private, smaller waterworks and 
designated facilities. However, to date the Ministry has not established any proactive 
inspection frequency requirements for them. Ontario’s Environmental Commissioner also 
expressed concern in his 2002 annual report that the Ministry did not inspect these facilities 
enough. He stated that they often serve susceptible populations, such as seniors and 
children, who are less resistant to contaminants and who face higher health risks than the 
general population. As the following chart summarizes, in 2002/03 the Ministry conducted 
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54 inspections on the 357 non-municipal, private water-treatment plants and 44 
inspections on the 1,119 small, private water-treatment plants and designated facilities. 


Waterworks Inspection Coverage, 2001/02—2002/03 
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Our analysis of ministry records also indicates that—except for the SWAT team that now 
conducts about one-quarter of all inspections—total inspection coverage is actually 


dropping. 


The following table illustrates this point. Since 1999/2000 the Ministry has increased the 
number of drinking water program inspectors by 57, bringing the total number of regional 
inspectors to 231 (an increase of 33%). However, over the same period, the number of 
inspections conducted actually dropped by 11% (from 4,182 to 3,707). Given the 
increased number of inspectors, this drop represents a decrease from an average of 24 
inspections completed annually per regional inspector in 1999/2000 to 16 inspections on 
average last year. The Ministry informed us that this can be attributed to the increase in 
length of time it now takes to conduct inspections. For example, a drinking water system 
inspection takes more than twice as long to complete under the current inspection process. 


Inspections Conducted by the Ministry 


Overall 





Source of data: Ministry of the Environment 
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Another issue is how inspectors are allocated across the province. The Ministry divides the 
province into five regions. However, we noted that only 37 of the 231 inspectors were 
assigned to the province’s central region, where more than 5 million Ontarians reside or 
almost half of Ontario’s population. By contrast, the northern region, which serves fewer 
than 1 million people, has been assigned 39 inspectors. We saw no formal evidence or 
analysis that supported the allocation of inspectors by region. Such an analysis would 
include factors such as the number and size of facilities to be inspected and travel distances. 


As stated earlier, inspectors are not currently using the Environet systems to plan or prioritize 
their work. We found that inspectors were not able to access the most accurate, complete, 
and timely information available, particularly regarding compliance history. Summary 
reports that use the information in Environet databases would improve the process of risk- 
based inspection planning; for example, reports to water inspectors could analyze past 
exceedances by facility, by seriousness, and by type. Accordingly, we urge the Ministry to 
accelerate the development of an Environet reporting application to provide the 
information that management and inspectors need. 


Recommendation 


To ensure inspection coverage is risk-based and that inspection resources are 
allocated most efficiently, the Ministry should: 


e develop Environet reports that analyze the state of Ontario’s environment 
and compliance with its regulations so that inspection resources can be 
allocated based on the greatest risks to human health; 


e re-assess waterworks inspection coverage to ensure more non-municipal 
waterworks are inspected; and 


¢ complete the development of a regime for laboratory inspections to ensure 
testing standards are being met and all Adverse Water Quality Incidents are 
reported promptly. 


Ministry Response 


The Ministry uses reports and information generated from across all program 
areas and databases, not just from Environet systems, to inform its work- 
planning and priority-setting activities. The Ministry recognizes that 
improvements to the resource allocation process are likely to come from a 
better ability to collect, integrate, and analyze data, particularly through our 
Environet strategy. As a first step, the Ministry is developing an information 
technology strategy to bring many drinking water databases together into one 
Environet-compatible network and data model. 








In addition to the Drinking Water Information System, the Ministry has received 

funding to develop systems such as the Laboratory and Waterworks Inspection 
System, which will contribute significantly to inspection and compliance 

“functions. When complete, this system will electronically assess waterworks 
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and provide a grading to identify and prioritize candidates for inspection. All 
aspects of the Ministry’s inspection and compliance strategy work together to 
target waterworks that may be non-compliant, most particularly in areas 
related to the protection of human health. 


Municipal drinking water systems provide drinking water to more than 80% of 
Ontario’s population. The Ministry now has a regulatory obligation to inspect 
all municipal drinking water systems on an annual basis. It has and will 
continue to fulfill this obligation. For non-municipal waterworks, the Ministry 
has now developed a structured risk-based program that responds to health- 
based Adverse Water Quality Incidents or complaints as necessary to protect 
public health and safety. 


The Ministry is making progress on the development and implementation of its 
laboratory licensing and inspection program. The Ministry’s inspection 
program (which ensures compliance with regulatory requirements) will be 
operational by October 1, 2003. Ontario’s accreditation program, which focuses 
on technical proficiency in the laboratories doing the testing, has been 
operational since 2000. 
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MINISTRY OF HEALTH AND LONG-TERM CARE 


3.09—Public Health Activity 





BACKGROUND 


The Ministry’s Public Health Branch, under the direction of the Chief Medical Officer of 
Health, is responsible for administering the Public Health Activity. The primary legislative 
authority governing the Activity is the Health Protection and Promotion Act, which provides 
for the organization and delivery of public health programs and services, the prevention of 
the spread of disease, and the promotion and protection of the health of the people of 
Ontario. Other legislation that plays a role in the Public Health Activity includes the 
Immunization of School Pupils Act, the Day Nurseries Act, and the Tobacco Control Act, 
1994. 


Public health services are primarily delivered through 37 local health units across the 
province. The Health Protection and Promotion Act provides the authority for establishing 
the local health units, which serve populations ranging in size from approximately 37,000 to 
2.5 million. Each health unit is governed by a local board of health and administered by a 
local medical officer of health who reports to the board of health. A board of health may be 
an autonomous corporation with representation from its constituent municipalities, or it 
may be part of a regional municipality. 


During the 2001/02 fiscal year, the Ministry provided approximately $222 million in 
transfer payments to local health units, primarily for the delivery of health programs and 
services mandated by the Ministry. Each local health unit is responsible for the delivery of 
public health programs and services in accordance with applicable legislation. In addition, 
local health units deal with emerging public health issues such as E. coli outbreaks, West 
Nile virus, and Severe Acute Respiratory Syndrome (SARS). 


AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of the Public Health Activity were to assess whether: 


* the Ministry’s expectations for public health were being met in a cost-effective manner; 
and 


¢ the Ministry had adequate processes in place to ensure that local health units were 
complying with applicable legislation and policies. 


Public Health Activity 217 





“VEM Section 3.09 | 


VFM Section 3.09 





a, 


Our audit focused primarily on the monitoring and funding of local health units for 
mandatory programs and services, surveillance, and vaccine-preventable diseases. We also 
followed up on the current status of recommendations made in our 1997 audit of the 


Public Health Activity. 


In conducting our audit, we reviewed relevant files and administrative policies and 
procedures, interviewed appropriate ministry staff, reviewed relevant literature, and 
researched the delivery of public health programs in other jurisdictions. While our audit 
focused primarily on the Ministry, we also visited a number of local health units to obtain 
their perspective on their responsibilities and surveyed the remaining local health units 
about specific activities. 


Our audit, which was substantially completed in March 2003, was conducted in 
accordance with the standards for assurance engagements, encompassing value for money 
and compliance, established by the Canadian Institute of Chartered Accountants and 
accordingly included such tests and other procedures as we considered necessary in the 
circumstances. The criteria used to conclude on our audit objectives were discussed with 
and agreed to by ministry management and related to systems, policies, and procedures that 
the Ministry should have in place. 


Towards the end of our audit, the Ministry and health service providers were coping with 
an outbreak of Severe Acute Respiratory Syndrome (SARS). Since our audit fieldwork was 
substantially completed before this outbreak occurred, our audit did not include work in 
this area. 


While the Ministry’s internal audit branch had performed some audit work on the Public 
Health Activity, this work related to areas outside the scope of our audit and therefore did 
not affect our audit. 


OVERALL AUDIT CONCLUSIONS 


The Ministry did not have adequate procedures to ensure that its expectations for public 
health were being met in a cost-effective manner. The importance of knowing that local 
health units are meeting the Ministry’s expectations for public health is significantly 
heightened in light of the emergence of new diseases such as West Nile virus and Severe 
Acute Respiratory Syndrome (SARS). The Ministry must be able to ensure that local health 
units respond quickly and properly to such diseases while continuing to minimize the health 
impact of existing diseases and continuing to provide other mandatory public health 
programs and services. 


Many of the issues and concerns raised in this audit were also identified in our 1997 audit 
of public health. In particular, we were concerned that the Ministry had not analyzed the 
extent to which individuals received differing levels of public health services or were exposed 
to greater levels of risk depending on where in Ontario they lived. Specifically, the Ministry 
had not analyzed whether funding for mandatory health programs and services was based 
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on assessed need or on a jurisdiction’s commitment or capacity to pay for the programs and 
services. In this regard, 2002 per capita funding for mandatory health programs and 


services, while averaging $37 for the province, ranged from approximately $23 per capita to 


$64 per capita among the 37 local health units. 


In addition, although the Ministry had introduced a questionnaire whereby local health 
units self-reported on their performance, the Ministry had conducted virtually no regular 
assessments of local health units in the last five years to determine whether the health units 
were complying with the guidelines for mandatory programs and services. Such assessments 
were recommended in the Report of the Walkerton Inquiry: The Events of May 2000 and 
Related Issues (Part One of the Walkerton Report). We also noted the following: 


None of the 33 local health units reporting information to the Ministry had conducted 
the necessary inspections of all of the food premises within their jurisdiction. In fact, 13 
of the 33 local health units had only conducted the required inspections for less than 
50% of the high-risk premises in their jurisdictions. Four local health units did not 


report their information. 


Seventeen out of 25 local health units that provided information to the Ministry 
reported that less than half of the high-risk food premises in their jurisdictions had food 
handlers who had the required training to help recognize and prevent risks associated 
with food-borne illnesses. 


In 2001, local health units only inspected approximately 60% of Ontario’s tobacco 
vendors to verify compliance with the provisions in the Mandatory Programs and 
Services Guidelines regarding sales to people under the age of 19. In addition, only 
60% of vendors that had been warned or charged for non-compliance within the 
previous two years received the required semi-annual inspections. 


Local health units reported that, in 2001, only 65% of individuals identified by 
Citizenship and Immigration Canada as requiring medical surveillance for tuberculosis 
were successfully contacted and managed in accordance with the Ministry’s Tuberculosis 
Control Protocol. 


Federal guidelines state that immigrants with inactive tuberculosis who are placed on 
medical surveillance should receive a complete medical examination, including an x-ray, 
after arriving in Canada. These individuals are required to obtain a letter from a local 
health unit verifying their compliance with federal requirements. However, according to 
the Ministry, the federal government only requires that the individuals contact a local 
health unit. Nine of the 21 local health units that provided letters indicated that they 
would do so as soon as the individual contacted them, regardless of whether they had 
had a physical examination or x-ray. Except in rare circumstances, an immigrant cannot 
be forced to have a physical examination or x-ray. Therefore, issuing letters based on 
contact alone reduces a local health unit’s ability to ensure compliance with federal 
guidelines and places the community at increased risk. 
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¢ The limited information the Ministry had with respect to immunization indicated that 
at least 14% of children had not had all required vaccinations by age seven. 


¢ The Ministry lacked accurate and timely information on communicable diseases, 
vaccine-associated adverse events (that is, adverse events such as illnesses occurring as a 
result of vaccinations), and immunization, limiting its ability to identify and take the 
necessary action to mitigate potential problems. 


¢ The Ministry had not yet developed a process to ensure that all local health units were 
conducting risk assessments and taking appropriate action to decrease the incidence of 


West Nile virus. 


DETAILED AUDIT OBSERVATIONS 
FUNDING 


The Health Protection and Promotion Act (Act) requires municipalities within the 
jurisdiction of a local health unit to cover the costs incurred by the local health unit in 
fulfilling its legislated responsibilities. The Ministry provides grants to help cover the costs of 
mandatory health programs and services (the mandatory programs). The Act authorizes the 
Ministry to develop and publish guidelines for the mandatory programs and requires each 
board of health to comply with the guidelines. 


In our 1997 Annual Report we had recommended that, “To ensure that funding for all 
mandatory public health programs is allocated equitably, the Ministry should expand the 
use of indicators of service costs and of the relative health needs of communities.” Allocating 
funds equitably—that is, based on assessed need—helps ensure that individuals with similar 
needs have access to similar services, regardless of where they live in the province. 


At the time of our 1997 audit, the Ministry had just initiated an “equity funding strategy” 
that was used then to ensure that any reduction in funding to public health units was 
applied in a rational manner using indicators of community health needs and service costs. 
At that time, the cost of the mandatory public health programs and services was shared by 
the Ministry and municipalities. Depending on the program, the province’s share ranged 
from 40% to 100% of the cost. In 1998, through Local Services Realignment, the funding 
and responsibility for the delivery of mandatory public health programs and services was 
transferred to municipalities. Since 1999, the Ministry has generally funded 50% of each 
local health unit’s costs, up to a maximum of 50% of the health unit’s locally approved 
budget for providing the mandatory programs and services detailed in the Ministry's 
guidelines. 


In 2001, the Public Health Branch established a Funding Allecaon Horeauls Working 
Group “to determine a methodology for allocating provincial grants to boards of health for 
the delivery of mandatory public health programs and services.” In its September 2001 
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report, the Working Group, which included representatives from the Association of Local 
Public Health Agencies and the Association of Municipalities of Ontario, stated that “the 
concept of funding based on demographic and health status is relatively well established in 
the health sector of other jurisdictions.” The Working Group recommended developing a 
model consisting of three funding components: core (or base), per capita, and needs, noting 
that “the implementation of an equitable funding model is difficult when there are two 
funders that operate under different rules.” Under the current funding model, the 
province's contribution is dependent on the locally approved budget. 


The Working Group also noted that, traditionally, provincial funding for public health has 
been allocated on the basis of cost-sharing of the locally approved board of health budget 
and that “consequently, provincial grants may be based not only on need, but also on local 
commitment to public health and capacity to pay for services from the municipal tax base.” 
The Working Group noted that in 2000, there was “slightly more than a three-fold 
difference in per capita funding between the lowest and highest funded boards.” In our 
1997 Annual Report, we also had noted that “over time, significant variations in funding 
have occurred, with per capita funding for some boards being three times the rate of that 


for other boards.” 


A recent analysis performed by the Ministry identified that significant variations in per 
capita funding for the mandatory programs still exist among local health units. While the 
2002 average per capita rate for the province was approximately $37, the rate ranged from 
approximately $23 to $64 among local health units. At the time of our audit, the Ministry 
had not analyzed whether these variations had resulted in differing levels of services for 
individuals with similar needs depending on where in Ontario they live. We also noted that 
public health programs in unorganized territories, which are 100% funded by the 
province, receive approximately the same amount of funding as they did in 1991 

($3 million). The Ministry has noted that in “most areas of the north, funding for 
unorganized areas, on a per capita basis, is well below the per capita rate in areas covered by 
municipalities.” 

The Ministry also had not defined what constitutes an “eligible expenditure”—in particular, 
which administrative and overhead costs are eligible—beyond the requirement that eligible 
expenditures be incurred to comply with the Mandatory Programs and Services Guidelines. 


While not required to do so, most of the local health units submitted to the Ministry in their 
budgets a breakdown by specific mandatory program or service. At year end, local health 
units are required to submit audited financial statements, but again were not required to 
provide a breakdown by mandatory program or service. In our current audit, we noted that 
there were some significant variances in the proportion of local health units’ funding 
allocations spent on particular mandatory programs or services. For example, based on 
budget submissions for 2002, expenditures by local health units for the Vaccine Preventable 
Diseases mandatory program ranged from 5% to 21% of each health unit’s funding for the 
mandatory programs. The Ministry had not investigated the reasons for these variances. 
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Recommendation 

To help it meet its objectives for the Public Health Activity, the Ministry should 

ensure that individuals with similar needs and risks receive a similar level of 

service regardless of where in the province they live. 








To help ensure that provincial funding is allocated on a consistent basis, the 
Ministry should provide clear guidance on what constitutes an eligible public 
health expenditure. 


Ministry Response 





The Ministry notes that service and funding levels across the province are 
related to the current legislative and funding framework. However, the Ministry 
will conduct an analysis of the comparative risks and needs in relation to levels 
of service currently delivered throughout the province. Based on the results of 
the analysis, the Ministry will consider options, including modification to the 
legislative and funding framework in order to rectify any inconsistencies in 
service levels in Ontario. 


The Ministry will further clarify definitions of eligible expenses to local health 
units as part of the roll-out of the new program-budgeting system. — | 





COMPLIANCE WITH LEGISLATION AND 
GUIDELINES 


The Mandatory Health Programs and Services Guidelines (Guidelines), which were last 
revised in December 1997, consist of three general standards and 14 program standards 
that set out minimum requirements for the mandatory programs. The general standards— 
dealing with equal access, health hazard investigation, and program planning and 
evaluation—outline considerations or activities to be undertaken in planning for the 
mandatory programs. The program standards—which are grouped under chronic diseases 
and injuries, family health, and infectious diseases—specify the minimum requirements “for 
each program in order to contribute to the achievement of stated, province-wide public 
health goals. These programs translate broad goal directions into objectives and targets and 
outline the activities that boards of health are required to undertake to achieve the stated 
objectives.” At the time of our audit, we were advised that the Ministry was reviewing six of 
the program standards. 


Program Objectives in the Guidelines 


We noted several problems with the program objectives in the Guidelines. First, many of the 
objectives need to be updated. For instance, the Guidelines refer to the Ministry's 1998 
“Determination of Tobacco Vendor Compliance Protocol,” yet the Protocol was revised in 
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2000. In addition, some objectives still refer to a year 2000 target date. We also noted that 
some program objectives do not have specific targets. For example, one objective under the 
Prevention of Chronic Diseases program is to increase the proportion of smoke-free homes 
by 2010; however, the objective does not include a target to be reached. We further noted 
that, in some instances, the Ministry did not have reliable data that could be used to 
determine whether boards of health were meeting objectives on a timely basis. For example, 
the Vaccine Preventable Diseases Program includes an objective to achieve, by the year 
2000, coverage rates of 95% for the immunization of children against certain diseases by 
their second birthday. However, because the Ministry does not gather immunization 
information for all children by their second birthday, it can only determine whether the 
objective was met using data provided when children enter a licensed daycare facility or in 
many cases when children enter school—three years after their second birthday. 


Assessment of Compliance 
Under the Act, the Minister of Health and Long-Term Care may assess whether local health 


units are providing or ensuring the provision of health programs and services in accordance 
with the Guidelines. In addition, Part One of the Walkerton Report, released in January 
2002 (the report was the result of the Walkerton Inquiry, established in June 2000 to 
investigate the water-borne E. Coli outbreak in Walkerton, Ontario), recommended that 
the Ministry conduct random assessments on a regular basis to ensure local health units are 
complying with the Guidelines. The report also stated that the Ministry should annually 
track trends in non-compliance in order to assess whether changes are required to the 
mandatory programs and whether resources require adjustment to ensure full compliance. 


Ministry staff informed us that, since 1998, only one assessment of a local health unit had 
been undertaken and that in March 2003, the Ministry began limited assessments of 
mandatory program areas at five local health units. 


When the Guidelines were revised in 1997, the Ministry estimated that it would take three 
years to achieve full compliance. In 1998, the Ministry initiated an annual Mandatory 
Programs Indicator Questionnaire (MPIQ), whereby local health units answered a series of 
questions related to the Guidelines. The Ministry uses their answers to assess whether 
program requirements are being met. At the time of our audit, the Ministry was in the 
process of reviewing the MPIQs covering the year 2001. 


We questioned the Ministry’s full reliance on the MPIQ as a basis for its assessment, as the 
MPIQ data consisted solely of local health units’ self-reported answers and the Ministry did 
not have any procedures in place for verifying the reliability of the information reported. In 
this regard, in 2000, the Mandatory Programs Measurement Working Group, comprising 
representatives from the Public Health Branch and Ontario’s Association of Local Public 
Health Agencies, recommended that the MPIQ be evaluated for its validity as a tool for 
assessing compliance with the mandatory programs. At the time of our audit, the 
recommended evaluation had not been conducted. 
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Based on its review of the completed MPIQs for the year 2000, the Ministry concluded 
that local health units were 78% compliant with the Guidelines. This was calculated by 
averaging the overall compliance rate for each of the MPIQ areas across the 37 local health 
units. However, we noted that this calculation was not a meaningful measure of compliance 
and was therefore not an indicator of the actual performance and overall effectiveness of 
public health programs across the province. Specifically, we noted the following weaknesses 
in the compliance calculation and the MPIQ itself. 


¢ The Ministry calculated overall compliance without considering the relative size of 
individual health units (the population served by the largest local health unit is over 
60 times that of the smallest health unit). 


¢ Compliance was assessed in absolute, “either/or” terms, rather than taking into account 
degrees of compliance. For instance, one health unit was about 10% compliant in a 
mandatory program area while another was 70% compliant, yet both were rated equally 
non-compliant. 


¢ The MPIQ did not elicit compliance data for all of the mandatory programs and 
services. For example, the Guidelines include an objective for a coverage rate of 95% for 
vaccinating children for hepatitis B by the end of grade 7, but the MPIQ did not 


address hepatitis B vaccination coverage rates. 


Some local health units had taken the initiative of using other measurement tools that may 
warrant consideration by the Ministry. For instance: 


¢ Fourteen of the 37 local health units have chosen to have their organizational and 
administrative functions—including program planning, implementation, monitoring, 
and evaluation—measured against the principles and standards that are peer-set by the 
Ontario Council on Community Health Accreditation. However, the Ministry was not 
receiving or requesting copies of the accreditation reports from participating health 
units. We noted that the Mandatory Programs Measurement Working Group also had 
indicated its support of accreditation as a validation of compliance with the mandatory 
programs. 


e The Ministry and other stakeholder groups have jointly been conducting pilot projects 
on benchmarking various public health areas such as universal influenza immunization 
and the inspection of food premises. 


The Ministry had not evaluated any of these initiatives to determine whether they effectively 
measure compliance with the Guidelines or whether they should be expanded to measure 
compliance across the province. 


Full-time Medical Officer of Health Requirement 


The Health Protection and Promotion Act requires local health units to employ the services 
of an appropriately trained medical officer of health. Part One of the Walkerton Report 
recommended that the Act be amended to require boards of health and the Minister to 
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expeditiously fill any vacant medical-officer-of-health positions with a full-time medical 
officer of health. In December 2002, the following amendment to the Act received royal 
assent: “If the position of medical officer of health of a board of health becomes vacant, the 
board of health and the Minister, acting in concert, shall work expeditiously towards filling 
the position with a full-time medical officer of health.” According to the Ministry, there is a 
national shortage of physicians with community medicine training to fill vacancies, and as of 
January 2003, there were eight boards of health without the mandated full-time medical 
officer of health. While there were individuals acting in the medical-officer-of-health 
position, according to the Ministry they may not have had all of the required qualifications 
for the position. At five boards, acting medical officers of health had occupied the position 


for over three years. 


At the end of our audit, the Ministry was advertising to attract potential candidates. We 
were also advised that the Ministry has identified community medicine as a priority specialty 
for the Ministry’s recruitment, retention, and retraining program. 








Recommendation 














To help ensure compliance with legislation and the Mandatory Health 
Programs and Services Guidelines, the Ministry should: 


e establish more valid measures for assessing the performance and overall 
effectiveness of public health programs and services delivered by local 
health units; 


¢ periodically verify the reliability of the compliance information reported by 
local health units; and 

¢ ensure that every local health unit has a full-time medical officer of health 
as required by legislation. 





Where local health units are using other measurement tools, such as 
accreditation, the Ministry should: 


¢ obtain any resulting reports and analysis; and 
e assess whether any of these tools should be used by all local health units. 





Ministry Response 


The Ministry will re-examine current measurement tools with the aim of 
implementing improvements to the performance measures. 


Subject to funding approvals, the Ministry will undertake an assessment 
program in order to verify compliance information reported by local health 
units. In addition to the limited assessments undertaken in 2002/03, the Public 
Health Branch is currently undertaking an assessment in one local health unit. 





The Ministry will continue its efforts to ensure that every local health unit has a 
full-time medical officer of health through promotion and expansion of the 5 
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Medical Officer of Health In Training bursary program and the national 
advertising campaign, subject to funding approval of the bursary program. 


The Ministry will consult with local health units, the Ontario Council on 
Community Health Accreditation, and other stakeholders on the acceptability 
of accreditation information as a source of input for the Ministry’s assessment 
of affected local health units. 








FOOD SAFETY 


The Guidelines state that the goal of the Food Safety program is “to improve the health of 
the population by reducing the incidence of food-borne illness.” The objectives of the 
program are to ensure that food is prepared, stored, and served in a manner consistent with 
accepted public health practices and “to stop the sale or distribution of food that is unfit for 
human consumption by reason of disease, adulteration, impurity or other cause.” 


Inspections of Food Premises 


The Guidelines require that local health units assess all food premises at least annually and 
determine their risk status (high, medium, or low) based on risk assessment criteria specified 
in the Ministry's Hazard Analysis Critical Control Point (HACCP) Protocol. Inspections are 
required at high-risk premises at least once every four months, at medium-risk premises at 
least once every six months, and at low-risk premises at least once a year. High-risk premises 
must also have an annual HACCP Protocol audit, which focuses on those steps in the food 
preparation process that can control or eliminate food safety hazards. Inspections focus on 
regulatory requirements such as basic sanitation. 


In our 1997 Annual Report, we recommended that the Ministry should determine whether 
local health units had fully implemented HACCP and take corrective action as necessary. 
The Ministry replied that it would be setting and monitoring program standards and would 
take action as appropriate to enforce the standards. In our current audit, we noted that, 
based on information reported to the Ministry for 2002, about 60% of local health units 
had completed less than half of the required annual HACCP Protocol audits. We also 
found that the Ministry had not reviewed the application of the HACCP risk-assessment 
criteria by the health units and therefore had no assurance that the assessed risk had been 
assigned to food premises in a consistent manner across the province. 


Based on our review of information submitted to the Ministry by 33 of the 37 local health 
units, we found that no local health units were conducting 100% of the required 
inspections for premises in their jurisdiction. In fact, in 2002, for food premises that had 


been assessed as high risk: 


° only six of the 33 local health units conducted the required inspections for more than 
90% of the premises; 
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* 14 conducted the required inspections for from 50% to 90% of the premises; and 


* 13 conducted the required inspections for less than 50% of the premises. 


A similar pattern of inspection coverage had occurred for medium- and low-risk premises. 






Recommendation 


To help minimize the risk to the public of food-borne illnesses, the Ministry 
should ensure that local public health units are conducting the required 
inspections and Hazard Analysis Critical Control Point Protocol audits of food 
premises to ascertain whether food premises are complying with acceptable 
public health practices. 












Ministry Response 






The Ministry will include the food safety mandatory program in the health unit 
assessment program, subject to funding approval of the assessment program. 





Food-handler Training 


Another component of ensuring food safety involves the proper training of food handlers. 
‘Training assists food handlers in recognizing and preventing risks associated with food-borne 
illnesses. 


In our 1997 Annual Report, we recommended that the Ministry determine whether local 
health units had fully implemented food-handler training. In January 1998, the Ministry 
issued the Food Handler Training Protocol, which specified the circumstances in which a 
trained food handler must be in food premises at all times. The Guidelines require that local 
health units ensure that food-handler training courses are provided in accordance with the 
Protocol to food handlers in high- and medium-risk food premises. Training requirements 
were to be fully implemented by 2002. 


The Ministry does not require the presence of trained food handlers at food premises of any 
assessed level of risk when fewer than three employees prepare food at the same time. 
However, the Ministry does not have any evidence that establishments with lower numbers 
of food handlers are less prone to food-borne illnesses. In comparison, British Columbia has 
legislated a requirement that all food service establishments have a person trained in food 
handling on site at all times. 


Several local health units indicated that they have had difficulty in requiring food handlers 
to be appropriately trained. In 2002, 17 out of the 25 local health units that provided 
information to the Ministry indicated that less than half of the high-risk premises in their 
jurisdiction had trained food handlers; 91% indicated that less than half of the medium-risk 
premises had trained food handlers. A number of local health units indicated that legislating 


these requirements would assist them in ensuring that food handlers are properly trained. 
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Recommendation 


To help minimize the risk to the public of food-borne illnesses, the Ministry 
should: 





e ensure that public health units are complying with food-handler-training 
requirements; 

e assess the risk of not requiring trained food handlers at food premises 
using fewer than three employees to prepare food; and 

e determine whether food-handler training should be legislated. 


Ministry Response 


The Ministry will continue to monitor local-health-unit compliance with the 
requirement to ensure the provision of food-handler training. In addition, the 
Ministry will promote the use of food-handler training programs through local 
health units and food industry trade organizations. 





The Ministry is supportive of having trained food handlers in all high- and 
medium-risk food premises regardless of the size of the operation. 


The Ministry will continue to seek approval for regulatory amendments that 
would require the presence of a certified food handler in all medium- and high- 


risk food premises in Ontario. | 





TOBACCO CONTROL 


The Ministry’s Chronic Disease Prevention Program, which includes tobacco control, is 
intended to reduce premature mortality and morbidity from preventable chronic diseases 
such as stroke, lung cancer, and oral cancer. The Tobacco Control Act, 1994 was enacted 
primarily to provide legislative support to efforts to reduce the rate of smoking and to 
protect Ontarians from second-hand smoke. 


The Guidelines’ objectives in this regard include: reducing tobacco use by teenagers and 
adults who smoke daily; increasing smoke-free public places and workplaces; and reducing 
tobacco sales to people under the age of 19. These objectives specifically state that the 
proportion of 12-to-19-year-olds who smoke daily was to be reduced to 10% by the year 
2005, and the proportion of vendors that sell tobacco to minors (persons under the age of 
19) was to be reduced to 10% by the year 2000. Statistics Canada reported that in the 
2000/01 fiscal year, about 11.4% of 12-to-19-year-olds in Ontario smoked daily. 


The Ontario Tobacco Research Unit (OTRU)—a ministry-funded organization that 
researches tobacco-related health problems in Ontario—noted that in 2001, 35% of 
Ontario students reported they were able to purchase tobacco without being asked to 
provide photo identification, a situation that had not changed since 1995. In December 
2002, OTRU noted that progress in restricting access to minors had been good in some 
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locales but was generally uneven and that compliance was generally not improving. 
However, information in the 2001 Mandatory Program Indicator Questionnaires (MPIQs) 
indicated that overall reported compliance in not selling to minors was 87%. While this 
indicates a relatively high success rate, we noted that the Ministry had not followed up on 
OTRU’s conclusion that compliance was not improving. In 2002, Health Canada 
commissioned a national evaluation into retailer behaviour towards certain youth-access-to- 
tobacco restrictions. The results, based on seven major cities in Ontario, indicated that the 
overall provincial rate of vendor compliance in not selling to minors was approximately 


TI: 


Inspections and Compliance Checks 


Local health units are required to inspect all tobacco vendors at least annually and perform 
follow-ups to ensure compliance with the Zobacco Control Act, 1994. For a sample of 
vendors, local health units are also required to perform compliance checks, where a test 
shopper who is 15 to 17 years of age attempts to purchase tobacco. Where inspections or 
compliance checks discover issues of non-compliance, warnings are issued. If the vendor is 
found to still be non-compliant on a follow-up visit, the vendor is charged. Charges may 
result in fines or a prohibition from selling tobacco. 


The Ministry communicates its requirements regarding the frequency of inspections and 
compliance checks through the Guidelines and its tobacco control protocols. However, 
even though the Ministry revised its Tobacco Vendor Compliance Protocol in 2000 to 
require most local health units to increase the number of test-shopper compliance checks of 
tobacco vendors, the Guidelines continue to refer to the 1998 protocol. We noted that, in 
many instances, local health units did not perform compliance checks for a sufficient 
number of vendors according to the 2000 protocol. 


Based on information submitted to the Ministry on their 2001 MPIQg, local health units’ 


tobacco vendor-inspection activities were not as rigorous as required. Specifically: 


¢ Only 10 of the 36 local health units that provided inspection information to the 
Ministry indicated that they inspected 100% of their vendors during the year. A total of 
approximately 40% of tobacco vendors across the province did not receive the annual 
inspection that is required under the Guidelines. 


¢ Only 60% of the tobacco vendors who were warned of non-compliance or who were 
charged within the previous two years received the required semi-annual inspections in 


2001. 


Enforcement 
In December 1998, the then-Minister of Health and Long-Term Care appointed an 


advisory panel of experts to recommend how Ontario could achieve more effective tobacco 


control results. At the time, vendors charged under the 7obacco Control Act, 1994 could be 
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prohibited from selling tobacco for six months or subject to fines under either that Act or 
the Provincial Offences Act. 


In its 1999 report, the Expert Panel on the Renewal of the Ontario Tobacco Strategy 
indicated nine areas that needed to be addressed by the government. These included 
enforcement resources, fines and penalties for non-compliant vendors, increased smoke-free 
spaces, and increased tobacco prices. The panel also recommended that the Act “should be 
amended to make it easier to prosecute vendors, and should be streamlined and clarified in 
order to prevent senior courts from diluting its intent.” In 1999, fines under the 7obacco 
Control Act, 1994 were increased. However, no other significant changes were made to the 


Act. 
The Ministry had not followed up with local health units to determine whether the 


increased fines had had any impact on compliance. Although the Ministry was maintaining 
a cumulative summary of charges and convictions resulting from violations of the Zobacco 


Control Act, 1994, it did not track trends. 


The Tobacco Control Act, 1994 states that a vendor is automatically prohibited from selling 
tobacco for six months if the same person is convicted twice within five years of selling 
tobacco to someone under the age of 19. To assist in obtaining prohibitions of the sale of 
tobacco when appropriate, the Ministry’s 1994 guide to the Act indicates that, if possible, 
the owner should be charged in addition to the sales clerk. However, 80% of the local 
health units we surveyed indicated that they encountered problems in effectively enforcing 


the Act, including difficulty in obtaining a prohibition of the sale of tobacco. 


The expert panel also recommended that the government require that indoor public places 
be 100% smoke free. This coincides with one of the objectives of the mandatory programs, 


which stipulates that the proportion of smoke-free public places and workplaces be 
increased to 100% by the year 2005. However, the Tobacco Control Act, 1994 delegates the 
responsibility for achieving this objective, in part, to municipalities. The Ontario Tobacco 
Research Unit concluded in its December 2002 report that this objective is unlikely to be 
achieved without provincial legislation. Many of the local health units who responded to 
our survey also indicated their support for provincial legislation to assist in achieving local 
goals for smoke-free premises and public spaces. 










Recommendation 


To improve tobacco control in Ontario and thereby help achieve the Ministry’s 
goal of reducing premature mortality and morbidity from preventable chronic 
diseases, the Ministry should: 


e ensure that local health units work towards the goal of reducing the 
number of minors having access to tobacco products by conducting the 
required number of inspections and compliance checks; and 


e determine whether changes to legislation would assist the Ministry and 
local health units in better meeting tobacco control objectives. 
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Ministry Response 


Random assessments of local health units by ministry staff regarding tobacco 
enforcement were started in March 2003. These assessments will help improve 
local health units’ compliance with program standards (including the number 
of compliance checks) outlined in the Mandatory Health Programs and 
Services Guidelines. 


The Ontario Tobacco Strategy is in the process of reviewing training needs of 
tobacco-enforcement staff in health units to further assist with tobacco 
enforcement and to better ensure compliance with the Mandatory Health 
Programs and Services Guidelines. 


The Public Health Branch will continue to seek approval for a comprehensive 
tobacco control strategy, including strengthening of the Tobacco Control Act, 
1994. 


TUBERCULOSIS CONTROL 


According to the World Health Organization, tuberculosis (TB) affects an estimated one- 
third of the world’s population, mostly in developing countries. Annually, an estimated 

8.4 million people are newly infected worldwide, and 1.9 million people die of the disease. 
In addition, the World Health Organization estimates that 50 million individuals are 


infected with strains of TB that are resistant to one or more common anti-I'B drugs. 


Infected individuals with a healthy immune system have only a 10% chance of developing 
active symptoms of IB. However, once someone develops active TB lung-infection 
symptoms, that person can spread the disease. People with inactive TB may take medication 
to help prevent the disease from becoming active. 


According to ministry data, the incidence of TB has been declining in Ontario, from 
6.8 per 100,000 people in 1997 to 5.8 per 100,000 people in 2001. One objective of the 
TB Control mandatory program is to reduce the annual incidence rate of active TB to 3.5 


per 100,000 people by the year 2005. 


Medical Surveillance 


Most of the cases of TB in Ontario occur among immigrants living in major urban areas. 
Individuals are at the highest risk of developing active TB within the first five years of their 
arrival in Canada. The foreign-born active and relapsed cases of TB in Ontario in 2001 
accounted for over 80% of all active and relapsed cases. 


All potential immigrants and some visitors to Canada are required to undergo a clinical 
examination. For individuals 11 years of age or older, this includes a chest x-ray. Individuals 
found to have active TB are not permitted to immigrate to Canada until they have 
completed treatment. Individuals with inactive TB lung infection are required to report to 
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their local public health unit for medical follow-up within 30 days of arriving in Canada. 
This is done to ensure that their TB has not become active and to determine whether 
medical treatment is appropriate. The Ministry provides local health units with information 
from Citizenship and Immigration Canada concerning these individuals. 


National guidelines for the investigation and follow-up of immigrants with inactive TB who 
are referred for medical surveillance are included in the Canadian TB Standards and state 
that these individuals should receive at least one complete medical evaluation. Important 
components of this evaluation include obtaining a comprehensive patient history and 
performing a physical examination and laboratory tests, including an x-ray. Such an 
evaluation is consistent with the requirements in the Ministry’s Tuberculosis Control 
Protocol, which provides guidance to local health units. The national guidelines also state 
that individuals who are placed under medical surveillance are required to provide to 
Citizenship and Immigration Canada evidence of compliance with the medical surveillance 
requirement from the local health unit to have their medical surveillance terms and 
conditions removed. According to Public Health Branch staff, the Citizenship and 
Immigration Canada requirement regarding evidence of compliance is met as soon as an 
individual contacts a local health unit. However, except in rare circumstances, an individual 
cannot be forced by the local health unit to have a physical examination or x-ray. 


We were informed that the Ministry does not keep track of the number of individuals that 
Citizenship and Immigration Canada has referred for medical surveillance. However, based 
on information reported by the local health units to the Ministry, we calculated that there 
were about 6,100 such individuals in 2001. Of the 21 local health units we surveyed that 
did issue the required evidence-of-compliance letters for Citizenship and Immigration 
Canada, nine indicated that they would provide the letters once individuals contacted them, 
in some cases even if only by telephone. Despite the requirement that health units continue 
with surveillance of those individuals, once a letter is provided an individual may choose not 
to have a physical examination or x-ray. The other 12 local health units indicated that they 
would provide the letter only after the individual had had a physical examination or x-ray. 
The national guidelines note that “non-compliance with prescribed treatment for inactive 
TB may be problematic for those on medical surveillance and has been significantly 
associated with the development of TB disease in refugees.” 


In our 1997 Annual Report, we recommended that the Ministry should improve its ability 
to track individuals under surveillance for inactive TB. At that time we noted that the Public 
Health Branch had indicated that approximately 35% of the individuals who were required 
to undergo medical surveillance for inactive TB by boards of health, including notifying the 
appropriate authorities of address changes, could not be followed up on due to missing or 
incorrect information such as a wrong address provided. Ministry staff also indicated at that 
time that Public Health Branch staff may be able to use OHIP’s Registered Persons Data 


Base to obtain the necessary information. 
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Information reported by local health units to the Ministry for the 2001 year indicated that 
only 65% of referred individuals were successfully contacted and managed by local health 
units in accordance with the Ministry’s Tuberculosis Control Protocol. We were advised that 
local health units were required to inform the Ministry of those individuals who could not 
be contacted. However, the Ministry had not determined whether local health units were 
fully complying with this requirement. Procedures had also not been implemented to utilize 
the Ontario Health Insurance Program’s (OHIP’s) Registered Persons Data Base to attempt 
to locate individuals who had not reported to a local health unit or had not undergone a 
physical examination and x-ray. 








= 
Recommendation 


To help reduce the incidence of active tuberculosis, the Ministry should 
enhance the effectiveness of medical surveillance by: 


e ensuring that local health units consistently and appropriately complete the 
medical surveillance of individuals with inactive tuberculosis, including 
ensuring that they have undergone a physical examination and x-ray; and 


¢ using all available sources of information, including the Ontario Health 
Insurance Program’s Registered Persons Data Base, to track those 
individuals under medical surveillance who were not successfully 
contacted and managed by local health units. 











Ministry Response 


The Ministry will confirm with local health units that, notwithstanding 
Citizenship and Immigration Canada’s limited requirement for persons on 
medical surveillance (that they make contact with the local public health 
authority), the follow-up by local health units in Ontario for persons on medical 
surveillance must be completed as outlined in the Tuberculosis Control 
Protocol (January 1998). This follow-up by local health units would require, as 
a minimum, that the person on medical surveillance has undergone a follow-up 
medical examination and chest x-ray in Canada. 


The Ministry has put processes in place to improve the effectiveness of 
medical surveillance. The Ministry will continue to work with Citizenship and 
Immigration Canada and the Ministry’s Registration and Claims Branch to 
determine how their respective data can be shared with local public health 
units. i, 











Contact Tracing 


The Ministry’s Tuberculosis Control Protocol requires that local health units locate and 
screen (trace) individuals who have been in contact with a person with active TB, in order to 
ensure that such individuals do not have active TB and to determine if they should be 
receiving treatment. 
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In our 1997 audit, we recommended that the Public Health Branch obtain additional 
information on the results of TB contact tracing by boards of health. The Ministry 
responded that a new information system for tracking reportable diseases was in early 
development and that additional information on individuals who have come in contact with 
a person with active TB would be included in the system. At the time of our current audit, 
such a system had not been put in place, and the Ministry’s information on the extent and 
results of contact tracing was still limited. In addition, ministry and local health unit staff 
informed us that, except under rare circumstances, they generally cannot force individuals 
who have come in contact with a person with active TB to be screened. We were informed 
that the Ministry is considering a federal/provincial/territorial initiative to implement an 
automated public health information system that would support public health case 
management. Such a system would also prove useful in cases of other communicable 
diseases. 








: 2 
| Recommendation 


To help monitor the effectiveness of tuberculosis control in reducing the risk 
of spreading active tuberculosis, the Ministry should obtain more complete 
information on the results of tuberculosis contact tracing by local health units. 


Ministry Response 





The Ministry will continue to work with Health Canada, other provinces and 
territories, and other stakeholders to implement the integrated Public Health 
Information System for the surveillance of communicable diseases. This 
system, when implemented, should enable the Ministry to obtain information 
es the results of contact tracing. 








Treatment 


The Tuberculosis Control mandatory program requires that local health units monitor 
patient adherence with prescribed drug regimens, including monitoring for the completion 
of therapy in accordance with the Ministry’s Tuberculosis Control Protocol. A patient's 
failure to comply with the drug regimens may result in a recurrence of TB that is drug 
resistant. 


In our 1997 audit, we noted that the Ministry was reviewing the need for directly observed 
therapy (DOT), whereby a trained individual directly observes the patient taking the 
required anti-T'B medication. The World Health Organization recommends DOT. In 
1998, the Ministry issued the Tuberculosis Control Protocol, which stated that DOT should 
be arranged for all patients with respiratory TB, unless their assessment indicated they are 
highly likely to comply with therapy. This is consistent with national standards. However, 
information reported by local health units to the Ministry for 2001 indicated that only 66% 
of patients requiring DOT actually underwent any direct observation. 
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Recommendation 


To help prevent the spread of drug-resistant tuberculosis, the Ministry should 
develop and implement strategies to better ensure that all patients actually 
complete the required treatment. 





Ministry Response 






The Ministry will continue to monitor local health unit activity with regard to 
the number of patients treated with Directly Observed Therapy. The Ministry 
will continue to review treatment completion data and will work with local 

health units to develop strategies to improve the treatment completion rate. 







VACCINE-PREVENTABLE DISEASES. 


The development of vaccines and immunization programs has almost eliminated some 
diseases and drastically reduced the impact of others. As international travel has increased, 
immunization and vaccination programs have been implemented on a wide scale to help 
protect individuals from contracting diseases, such as measles, from infected travellers from 
foreign countries where the diseases are still common. 


The goal of the Ministry's Vaccine Preventable Disease mandatory program is “to reduce the 
incidence of vaccine preventable diseases.” Immunization is widely considered to be a cost- 
effective health intervention for achieving this goal, leading to improved health, reduced 
suffering, and fewer premature deaths. Ontario’s Association of Local Public Health 
Agencies (ALPHA) has stated that “effective vaccination programs have repeatedly proved 
to be one of the most cost-beneficial components of the health care system.” The Ministry 
provides certain vaccines free of charge to local health units and physicians. In the 2002/03 
fiscal year, the Ministry spent $67.7 million on vaccines. 


Vaccines Covered 


The range of immunizing agents available continues to expand as new vaccines are licensed 
or as currently available vaccines are improved or modified. The Ministry currently covers 
the cost of routine vaccinations against measles, mumps, rubella, diphtheria, tetanus, polio, 
pertussis, haemophilus influenza type B, influenza, and hepatitis B. 


The National Advisory Committee on Immunization, an expert advisory board to Health 
Canada, makes recommendations on the use of vaccines in Canada. It recommends routine 
vaccination against varicella (chickenpox) and certain types of bacterial infections that can 
cause serious illnesses such as pneumococcal and meningococcal disease (meningitis). Some 
provinces in Canada offer these vaccines to their population at no cost. For example, British 
Columbia and Alberta offer the meningococcal vaccine at no cost, and PEI and Alberta 
offer the chickenpox vaccine at no cost. In Ontario at the time of our audit, the chickenpox 
vaccine was not being provided on a routine basis, the pneumococcal vaccine was being 
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covered only for high-risk persons, including persons over 65 years old, and meningococcal 
vaccine was being provided only to individuals in close contact with individuals who have a 
vaccine-preventable strain of the disease. We noted that ALPHA has stated that “by failing 
to fund new licensed and effective vaccines, the province is displaying an incomplete 
commitment to the goal of the program that its own Ministry of Health has set.” 





Recommendation 


To help reduce the incidence of vaccine-preventable diseases, the Ministry 
should ensure that other vaccines recommended by the National Advisory 
Committee on Immunization are added to Ontario’s routine immunization 
program unless sound reasons exist for not including the recommended 
vaccines. 


Ministry Response 


The Ministry will continue to review all new vaccines recommended by the 
National Advisory Committee on Immunization and to develop and propose 
implementation options (including costing) for the introduction of these 

| vaccines in Ontario. 

















Immunization 


IMMUNIZATION OF CHILDREN 


The immunization of children is primarily governed by the Immunization of School Pupils 
Act, the Day Nurseries Act, and the Guidelines. The mmunization of School Pupils Act and 
the Day Nurseries Act require that all school-aged children and children attending licensed 
child care facilities in Ontario be vaccinated against certain diseases unless they have an 
exemption for medical reasons, statement of conscience, or religious beliefs. 


The requirements of the Guidelines and legislation vary with respect to the information 
about children to be provided to local health units. For example: 


¢ The Guidelines require that a record of each child’s vaccination status or exemption be 
on file with the local health unit. If no record is on file, local medical officers of health 


can have the child suspended from school. 


¢ Although not required to under the Jmmunization of School Pupils Act, local health units 
do obtain the vaccination status of children regarding pertussis (whooping cough) and 
haemophilus influenza type B—vaccinations that are required for children entering a 
day nursery. 


¢ The Education Act requires that public schools provide the local health unit with 
information on the students enrolled at the school, including the student's name, 
address, date of birth, and parent’s/guardian’s name. 
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Local health units enter the immunization status of children into the local Immunization 
Records Information System (IRIS), for local and provincial tracking. However, the 
Ministry cannot access immunization information on children who were entered without an 
OHIP number and must rely on reports from local health units for this information. While 
local health units have the right to collect OHIP numbers, schools are not required to 
provide them. As a result, it is difficult for local health units to ensure that the records of 
children entered without a number are not duplicated (for example, if the children change 
schools or local health units). IRIS information may also be incomplete for pre-school 
children not in licensed child care facilities and for children in private schools, since the 
Education Act does not require private schools to provide the local health unit with a list of 
students. In these cases, it is up to parents to contact the health unit with information on 
their children. Consequently, the IRIS does not have sufficient data to provide the Ministry 
with a complete picture of the immunization status of children, and therefore the Ministry 
relies on the information self-reported by local health units through the MPIQ process or 
provided in response to a ministry request. The 2001 MPIQs indicated that at least 14% of 
children entered in IRIS had not had all required immunizations by age seven. 


IMMUNIZATION REGISTRY 


Immunization registries are confidential computerized information systems in which 
vaccination data about individuals are stored. The registries consolidate all the vaccination 
records from an individual’s health care providers, identify individuals who are due or late 
for vaccinations, generate reminder notices to ensure that individuals are vaccinated 
appropriately, and identify service providers or geographic areas with low vaccination 
coverage. 


In Ontario, vaccines are primarily administered by doctors but may also be administered 
through various other means, including public health clinics. However, physicians and 
other administering groups are not required to report to local health units on vaccines 
administered either at the time of immunization or at any later time. Instead, as already 
mentioned, Ontario’s current legislation places the onus on parents to report the 
immunization status of their children to local health units. Since parents generally do not 
report immunization results until a child enters a licensed day care or school, immunization 
information for pre-school children not in licensed facilities is incomplete at best. In 
addition, because the information is generally reported a number of years after the child was 
vaccinated, children’s records may be inaccurate. 


In our 1997 Annual Report, we recommended that the Ministry assess the feasibility of 
modifying existing systems or developing appropriate ones to capture necessary 
immunization information. The Ministry responded that it was working on improving 
monitoring systems to enable better assessment of immunization coverage in the population. 
However, no such system had been implemented at the time of this audit. 


We were informed that the Ministry was considering a federal/provincial/territorial initiative 
to implement an automated public health information system that would support public 
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health case management and the sharing of communicable disease surveillance and 
immunization information. The Ministry recognized that this system would need to be 
modified to provide all of the information that the Ministry needs. 





Recommendation 


To help achieve its goal of reducing the incidence of vaccine-preventable 
diseases, the Ministry should more effectively monitor the immunization status 
of children to ensure that all school-aged children have had the required 
vaccinations. To this end, the Ministry should ensure that it has an 
immunization registry that provides complete, accurate, and timely 
immunization information. : 





Ministry Response 


The Ministry supports the National Immunization Strategy by working with the 
Canadian Immunization Registry Network to assist in the development of a 
core data set and standards. In addition, the Ministry supports efforts to 
develop an immunization registry that effectively monitors the immunization 


status of children in Ontario. 
fee 











Influenza Vaccine 


Prior to the 2000/01 fiscal year, only high-risk individuals, such as seniors and health care 
workers, were eligible to receive the influenza vaccine at no charge. Since then, the Ministry 
has provided the influenza vaccine free to all Ontarians through its universal influenza 
immunization program. 


The immunization program’s main objectives are to decrease the number of cases and 
severity of influenza and to reduce the impact of influenza on emergency room visits and 
other areas of the health care system. However, since the Ministry lacks data on the number 
of vaccines actually administered, it could only estimate that 26% to 45% of Ontarians were 
immunized against influenza during each of the 2000/01 and 2001/02 influenza seasons. 
(The variation in the estimate is due to differences in the estimating methods used and 
assumptions made.) 


The Centers for Disease Control and Prevention—a U.S. government agency—states that 
the “flu vaccine is the single most effective means for preventing infection with the flu virus 
and the complications associated with flu.” Yet Ontario is the only jurisdiction in North 
America to have a universal influenza immunization program. Therefore, assessing whether 
the influenza vaccine is meeting all program objectives is especially relevant. Such an 
evaluation is also important because the cost per dose of the influenza vaccine has almost 
doubled since the program started and Ontario spent $19.6 million on the vaccine in the 
2002/03 fiscal year. In the 2000/01 fiscal year, the Ministry established a committee—with 
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representation from the Institute of Clinical Evaluative Sciences, the Public Health 
Research, Education and Development group, and local health units—to provide advice 
regarding the evaluation activities that could be conducted. The Ministry noted in this 
regard that, since influenza seasons vary in severity, a multi-year outcome evaluation is 
required. 


While various stakeholders have examined the delivery of the program, as of March 
2003—the third year of the program—the Ministry had not evaluated whether, as a result 
of the program, the number of cases and the severity of influenza had decreased and the 
impact of influenza on emergency room visits and other areas of the health care system had 


been reduced. 






Recommendation 





To help determine the effectiveness of the universal influenza immunization 
program, the Ministry should evaluate whether the program is meeting its 
objectives of decreasing the number of cases and severity of influenza and 
reducing the impact of influenza on emergency room visits and other areas of 
the health care system. 











Ministry Response 


There are plans by the Ministry to reconvene the Universal Influenza 
Immunization Program Evaluation Planning Group to provide advice regarding 
the evaluation activities that could be conducted in the long term using 
cumulative data. 









Vaccine Wastage 


The Ontario Government Pharmacy and Medical Supply Service (OGPMSS) buys 
vaccines and distributes them free of charge to local health units, who may retain some for 
their own programs and distribute the rest to doctors and other vaccine-administering 
groups such as health care facilities. In Toronto, doctors, as well as other administering 


groups, receive their vaccines directly from the OGPMSS. 


Although a certain amount of vaccine wastage, mishandling, and over-ordering is inherent 
to any vaccine distribution process, one objective of the Vaccine Preventable Diseases 
mandatory program is to minimize wastage of provincially funded vaccines to 5% or less. 
The Guidelines require that local health units optimize vaccine use by following the 
Ministry’s Vaccine Distribution, Storage and Handling Protocol, which focuses on ensuring 
vaccine potency and reducing spoilage. 


In order for the Ministry to monitor vaccine use and wastage, it requires complete and 
accurate information on vaccines distributed, administered, returned, and wasted due to 
expiration or spoilage. The OGPMSS distributes vaccines on the basis of orders received 
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from local health units and Toronto doctors and tracks the quantities distributed. If an order 
size seems unusual, the OGPMSS may check it against past orders and may ask for 
clarification. The OGPMSS also requires that unused or spoiled vaccines be returned. 


Local health units and Toronto doctors are requested to complete a vaccine return record 
indicating the reason the vaccine is being returned and, in the case of local health units, 
whether the vaccine can be redistributed. Returned vaccines may be redistributed if they 
were properly stored and have not expired. All returns from doctors are considered wastage, 
because the vaccine may have spoiled due to improper storage. 


However, although the Ministry has information on vaccines distributed and some 
information on vaccines returned, information is not available on the amount of vaccine 
actually administered to patients. Without this information, the Ministry is unable to 
accurately determine the overall amount of vaccine wasted and the possible reasons why. 


We note that OGPMSS does report on vaccine wastage using the information available to 
it—it reported wastage of about 9% of the vaccines distributed to local health units and of 
less than 1% of those distributed to Toronto doctors in 2002. However, this reporting is 
incomplete because it excludes returns from doctors that are not accompanied by a return 
record—and we were informed that doctors often do not complete the return records or 
may otherwise dispose of the vaccine. In addition, while the OGPMSS’s tracking system can 
produce wastage reports by individual local health unit, it is unable to produce reports on 
wastage by individual Toronto doctor, limiting the Ministry’s ability to determine the extent 
of wasted vaccines and to take appropriate action. 


Recommendation 


To help limit vaccine wastage, the Ministry should obtain accurate and 
complete information about vaccine wastage and take appropriate action to 
reduce wastage. 


Ministry Response 


The Ministry’s Public Health Branch is in discussions with the Ontario 
Government Pharmacy and Medical Supply Service on this matter. The two will 
work together to streamline and improve the reporting and tracking of 
vaccines distributed, used, and wasted. The Ministry will remind doctors about 
the importance of completing and returning records for accounting purposes 
and of reducing vaccine wastage. 








WEST NILE VIRUS CONTROL 


West Nile virus (W/Nv) was first confirmed in North America in 1999 and in Ontario in 
2001. The first human cases in Ontario occurred in the summer of 2002. WNyv is carried 
by mosquitoes and affects birds and mammals, including people. Studies indicate that most 
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persons bitten by an infected mosquito will have no symptoms; however, approximately 
20% of those infected will develop a mild illness (for example, West Nile fever), and 1% 
develop a serious illness. During the 2002/03 fiscal year, the Ministry funded local health 
units with approximately $1.8 million for WNv activities. 


Since each of the 37 local health units in Ontario has a high degree of autonomy in 
protecting the health of its community, there exists the possibility of many different 
responses to W Ny. However, given that WNv is a provincial issue, in that WNv can spread 
across the province, there is a need for some consistency in responses. The Ministry has 


provided guidance to local health units, including the following: 


¢ In August 2000, the Ministry sent local health units a document that addressed the 
potential arrival of WNv in Ontario. The document described surveillance plans (for 
example, testing birds, mammals—including humans—and mosquitoes for WNv) and 
prevention, response, and control issues. Subsequent updated documents were sent to 
local health units in May 2001 and February 2002 and included the importance of 


public education to help reduce mosquito-breeding areas. 


¢ On March 22, 2003, the Ministry announced a seven-point action plan to combat 
WNv. This action plan included expanding laboratory services in Ontario to test for 
WNy, providing funding to local health units for insecticides, and enhancing 
surveillance by making WNv a reportable disease. 


¢ On March 27, 2003, the Ministry sent local health units the WNv Preparedness and 
Prevention Plan for Ontario 2003. This was the “field guide” to the seven-point action 
plan and provided the “planning basis for an Ontario approach to the prevention and 
control of West Nile virus across the province by the health units.” While this Plan 
covered a wide range of areas, it did not state when local health units should consider 
the use of insecticides. Instead, the Plan stated that, prior to using insecticides, local 
health units are required to conduct their own risk assessments, which should include 
factors such as community attitudes towards the risks posed by WNyv versus the likely 
benefits and risks of using insecticides. 


Notwithstanding this ministry guidance, most of the 37 local health units had to conduct 
their own research to determine best practices for when to use insecticides. In fact, many of 
the local health units we surveyed in April 2003 indicated that additional and more timely 
guidance on when to use insecticides was needed, and in 2002 none of the local health 
units carried out any insecticiding at all. 


Effective May 31, 2003, the Control of West Nile Virus regulation was made under the 
Health Protection and Promotion Act, requiring local medical officers of health to conduct 
risk assessments in accordance with the West Nile Virus Preparedness and Prevention Plan 
for Ontario 2003 (which was revised to reflect the new regulation). Local medical officers of 
health may give notice to their respective municipalities of any action required. In addition, 


the regulation provides guidance to local medical officers of health on insecticide use. 
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As of May 1, 2003, West Nile virus illnesses became a reportable disease, whereby 
physicians are required to report the specific diagnosis of WNy to the local health unit. 
According to the Ministry, the initial intent was to then have the cases recorded on the 
Ministry’s Reportable Diseases Information System (RDIS). Previously, WNv was reportable 
on RDIS only as a cause of viral encephalitis/meningitis, which could also be caused by 
other viruses. However, as of May 2003, there was still no electronic system in place to 
enable more timely reporting of all cases of WNv to the Public Health Branch, though as an 
interim step, the Ministry has requested local health units to manually report information 
on all probable and confirmed human cases of WNy. 


Recommendation | 


To facilitate an effective response to West Nile virus by local health units, the 
Ministry should ensure that: 





e local health units comply with the Control of West Nile virus regulation and 
other guidance provided by the Ministry, including conducting risk 
assessments; 


¢ local health units carry out West Nile virus interventions in a cost-effective 
manner based on the results of local risk assessments; and 


e there is an electronic system in place to record and report all cases of the 
West Nile virus on a timely basis. 


Ministry Response 


The Ministry will ensure compliance with the Control of West Nile virus 

regulation by: 

e reviewing board-of-health budget submissions for West Nile virus 
prevention and control; 

e holding weekly teleconferences with local health units; and 

e requiring post-season report-backs from boards of health. 





With regard to the cost effectiveness of interventions by local health units at 
the provincial level, the Ministry will also flag cost-effectiveness issues that will 
require working with Health Canada and the Ministry of the Environment. 


The Ministry is implementing the integrated Public Health Information System 
for all communicable diseases, including West Nile virus; however, the 
automated application will not be available until next year’s West Nile virus 
season. In addition, the Ministry has developed a process for electronic data 
transfer of laboratory test results of human West Nile virus testing to the 
Ministry, local health units, and Canadian Blood Services. This process will be 
beeke shortly. : 
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INFORMATION SYSTEMS 
Public Health Information Strategy 


In October 2000, the Ministry, in conjunction with a consulting firm, prepared a Public 
Health Information and Information Technology Strategic Plan. The Plan presented an 
overall information technology strategy for public health. However, at the time of our audit 
it had generally not been implemented. 


The Plan also identified a large number of systems that have been developed independently 
among the 37 local health units, primarily in areas where ministry-supported systems were 
inadequate or non-existent. The Plan noted that the sharing of information between the 
local health units and the Ministry was limited and that “current legislation and technology 
infrastructure limits sharing between the health units themselves.” 


The development of independent systems is a concern, as it could hinder the integration of 
public health information across the province, possibly resulting in the loss of timely, 
important information needed for public health interventions and for prevention activities. 
It is also a concern because of the duplication of effort, costs, and time associated with 
independently developed information systems. 


Surveillance Systems 


Health surveillance is the ongoing collection, analysis, and interpretation of information that 
can be used to plan and manage efforts to control diseases. This includes information that 
assists in controlling outbreaks, making informed resource allocation decisions, and 
developing or changing public health policies and programs to make them more effective. 
Currently, the Public Health Branch supports two surveillance systems: the Reportable 
Diseases Information System (RDIS)—for communicable diseases and vaccine-associated 
adverse events (such as illnesses occurring as a result of vaccination)—and the Immunization 
Records Information System (IRIS) for immunization. 


In our 1997 Annual Report we noted that the Ministry indicated that it planned to replace 
RDIS with an improved system. However, this has not happened, even though the 
Ministry's October 2000 Strategic Plan noted that RDIS “was developed in the late 1980s 
with technology that today is extremely outdated, proprietary, and very costly to maintain 
and support.” It further stated that, “one public health role is to analyze health surveillance 
data to create public health policy and to prioritize and amend public health programs. 
Much of the information required to provide this analysis is either unavailable or of 
questionable quality.” 


The only information a local health unit can access on a timely basis is information 
pertaining to its own jurisdiction. This may limit a health unit’s ability to manage fast- 
spreading outbreaks that may have occurred in other jurisdictions in Ontario. In addition, 
because local health units generally send communicable diseases data to the Ministry only 
on a weekly basis, cross-jurisdiction information may not be readily available at the Ministry 
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on a timely basis. Also, if local health units are behind in entering data into the systems, the 
information at the Ministry may be incomplete or inaccurate. 


As mentioned previously, we were informed that the Ministry was considering a federal/ 
provincial/territorial initiative to implement an automated public health information system 
that would support public health case management and the sharing of communicable 
disease surveillance and immunization information. We were also informed that there were 
plans to pilot this system in Ontario. 


In seeking improvements in its surveillance systems, the Ministry in 2002 participated in a 
health surveillance project as part of the Network for Health Surveillance in Canada 
initiatives—a collaborative federal/provincial/territorial effort to build the capacity for 
specific types of health surveillance at local, provincial, and national levels by, for example, 
developing information systems to support public health practice and health surveillance. 
The Network’s Communicable Disease Surveillance Sub-Group, which “is developing a 
long-term strategy for the surveillance of communicable diseases,” has also been focusing on 
developing data standards and data definitions for information collected on communicable 
diseases, immunization, and vaccine-associated adverse events. 


In assessing Ontario’s status as part of the health surveillance project, problems were noted 
with both RDIS and IRIS. For example, it was noted in the project that changes to data in 
both systems were not being tracked, making it difficult to know what changes had 
occurred, who had made the changes, and when the changes had occurred. In addition, 
many local health units were not collecting certain types of data (for example, laboratory 
slips and school lists) electronically and had to manually enter the data into the information 
systems. 


Recommendation 


To help ensure that timely, consistent, and integrated information is available to 
deliver public health services across the province, the Ministry should 
implement, either in conjunction with the federal/provincial/territorial initiative 
to implement an automated public health information system or independently, 
an adequate public health surveillance system for communicable diseases and 
immunization. 





Ministry Response 


The Ministry is proceeding to implement a province-wide, integrated Public 
Health Information System for the purpose of surveillance, case management, 
and reporting of communicable diseases in Ontario. This information system is 
available to all federal, provincial, and territorial jurisdictions through the 
Canadian Integrated Public Health Surveillance Collaborative. 
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MINISTRY OF TRAINING, COLLEGES 
AND UNIVERSITIES 


3.10—Ontario Student 
Assistance Program 





BACKGROUND 


The Ontario Student Assistance Program (OSAP) is a federally and provincially funded 
program that provides needs-based financial assistance to full-time students to enable them 
to attend an approved postsecondary school. The objective of OSAP is to help students 
from lower-income families meet the costs of postsecondary education so that all qualified 
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students can have access to postsecondary education. OSAP supplements the financial 
resources that students or, if applicable, their families are expected to contribute. 


Under the Ministry of Training, Colleges and Universities Act and related regulations, 
provincial financial assistance to students is provided primarily by loans under the Ontario 
Student Loans (OSL) program. The Ministry also administers the Canada Student Loan 
(CSL) program and the Canada Millennium Bursary (CMB) on behalf of the federal 


gover nment. 


The amount of loan funding students may receive is based on their financial need as 
calculated by the Ministry, but assistance cannot exceed $275 for each week of study in the 
case of single dependent or independent students and $500 per week for married or sole- 
support students. 


The assistance takes the form of loan guarantees, loan forgiveness grants, interest subsidies 
while students are attending school, interest relief during the repayment stage, and various 
need- and merit-based bursaries and scholarships (such as Aiming for the Top and Ontario 


Graduate Scholarships). 


Until July 31, 2001, financial institutions provided Ontario Student Loans directly to 
students under terms prescribed by provincial legislation. Effective August 1, 2001, two 
service providers selected by the federal government—one for publicly funded colleges and 
universities and one for private postsecondary schools—began administering all the loans. 
These providers issue loans directly to Ontario students using funds provided by the 
Government of Canada and by the province through the Ontario Student Loan Trust. 
Although students are expected to repay their loans after leaving school, the Ministry pays 
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the interest on their student loan balances while the students attend school. The Ministry 
also guarantees repayment of the OSL principal should students default on their loans. 


The Ontario Student Opportunity Grant (OSOG) was introduced in 1998/99 to replace 
the Loan Forgiveness Program. An OSOG limits the amount of debt that a student will be 
required to repay to $7,000 for two terms of study or $10,500 for three terms. OSOGs are 
paid directly to the Ontario Student Loan Trust after the completion of each academic year, 
reducing the student loan payable at that time. 


For the 2002/03 fiscal year, provincial OSAP expenditures totalled $356 million, which 
was allocated as outlined in the following chart. 


OSAP Expenditures, 2002/03 Fiscal Year 
($ million) 
Payments to students 
for other bursaries Interest on loans 
and scholarships $28.8 
$90.1' 





Administration costs 
$16 7 









Ontario Student 
Opportunity Grants or 


Claims for loan forgiveness 
defaulted loans $125.1 
$96 


Payments to Banks and Services Providers ($249.9) 


"$30.9 million of this $90.1 million is recovered from the federal government. 


? Administration costs include $10.2 million in purchased services, much of which relate to computer system upgrades 
and payments to the federal government for loan administration by the service providers. 


Source of data: Ministry of Training, Colleges and Universities 


In addition to OSAP, other sources of financial assistance are available to eligible students in 
Ontario. Public postsecondary schools also offer scholarships and bursaries paid from their 
own funds or from Ontario Student Opportunity Trust Funds, $300 million of which were 
provided by the Ministry in previous years as an endowment to the schools. 


All OSAP-designated schools have a Financial Aid Office that serves the school’s students. 


AUDIT OBJECTIVES AND SCOPE 


The objectives of our audit of OSAP were to assess whether key financial and management 
systems and procedures are sufficient to ensure that: 


¢ the program is delivered economically, efficiently, and in accordance with stated goals 
and objectives, legislated authority, and approved policies and guidelines; and 
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* effectiveness in achieving stated goals and objectives is reliably measured and reported. 


Our audit was performed in accordance with the standards for assurance engagements, 
encompassing value for money and compliance, established by the Canadian Institute of 
Chartered Accountants, and accordingly included such tests and other procedures as we 
considered necessary in the circumstances. 


Our audit was conducted primarily at the Ministry’s Student Support Branch and included 
examining controls over application processing and eligibility, in-study account 
management, OSOGs (loan forgiveness grants), and loan collection and default 
management. In addition, we visited several Financial Aid Offices at postsecondary schools 
to interview staff and examine documentation supporting selected loan transactions and 
their administrative procedures. We also researched data on student financial aid programs 
in other jurisdictions. 


The audit criteria used to conclude on our audit objectives were discussed with and agreed 
to by senior ministry management and related to systems, policies, and procedures that the 
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Ministry should have in place. 


We conducted our audit work during the period of December 2002 to April 2003 with an 
emphasis on transactions and activities conducted in 2002/03. Where appropriate, we 
extended our analysis of transactions to the previous two fiscal years. 


The Ministry’s Internal Audit Services Branch had been working with counterparts in the 
federal government to document, review, and strengthen controls over the new service 
providers under contract with the federal government, and to establish external audit 
arrangements for these providers that address risks to Ontario. Internal Audit had also 
recently completed an IT security audit of the OSAP information systems. Accordingly, we 
significantly reduced the scope of our work in these areas. The Internal Audit Services 
Branch also assisted us in performing several detailed analyses of data maintained on the 
OSAP information systems. 


We also tested controls over two scholarship programs—the Aiming for the Top and the 
Ontario Graduate Scholarships—and reviewed the accountability and reporting 
requirements established for the Ontario Student Opportunity Trust Funds provided by the 
Ministry but managed by publicly funded postsecondary schools. Our observations and 
recommendations with respect to this work were not significant enough to warrant 
inclusion in this report but were communicated in a letter to the Ministry. 


OVERALL AUDIT CONCLUSIONS 


Since our last audit in 1997 the Ministry has taken action to address a number of our 
recommendations and significantly improve the overall administration of the program. For 
example, application processing has become much more efficient since the introduction of 
on-line application processing, and procedures to ensure that the Financial Aid Offices in 





Ontario Student Assistance Program 247 





| VEM Section 3.10 | 





t 





postsecondary schools meet service delivery and program requirements have been 
enhanced. While systems and procedures to ensure that the program is delivered 
economically, efficiently, and in compliance with requirements have improved since our last 
audit, there are several areas where further action is required: 


¢ The Ministry had not adequately investigated the results of its efforts to verify incomes 
reported on OSAP applications against federal government income tax information in 
order to determine the risk of undetected abuse. For example, it had not determined 
the number of students and supporting individuals whose reported income could not 
be verified over several years. 


¢ The Ministry has paid about $2 million more annually in interest and risked at least 
$6 million more in annual default costs than it should have because: 


- the majority of each loan advance is paid out early in the study period even though 
the study period can be up to a year long; 


- many postsecondary schools do not have adequate or timely processes for 
monitoring and recording changes in students’ academic status that should have 
reduced their assistance entitlement; and 


- it requires prompt repayment of loan overpayments only in cases where a student 
has significantly under-reported income to cause the overpayment. 


¢ The Ministry paid loan forgiveness grants to students who did not qualify for such 
grants (students who either had reduced their course load to part-time status or 
appeared not to have attempted to complete their studies) because the students’ schools 


had not updated OSAP records to reflect these changes. 


¢ Our own analysis to identify instances and patterns of reporting errors or abuse by 
students revealed unlikely circumstances that the Ministry ought to have questioned, 
such as students or parents reporting an increase of three or more dependent children 
from the previous yeat. 


¢ The Ministry could further reduce the cost of defaulted student loans by several million 
dollars by making greater use of default management practices successfully employed in 
other jurisdictions. It had also not referred at least 60,000 additional defaulted loans on 
which collection efforts had been unsuccessful to the Canada Customs and Revenue 
Agency, which would collect the outstanding amounts from any future income tax 


refunds claimed by the debtors. 


e The Ministry had made insufficient efforts to measure and report on the program’s 
effectiveness. Loan default rates were the only published performance measure for the 
program. While the Ministry had been working with the federal government to develop 
more comprehensive performance measures for student assistance programs across 
Canada, no firm target date had been established for finalizing the measures or for 
collecting and reporting program results. 
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DETAILED AUDIT OBSERVATIONS 


DETERMINING ELIGIBILITY AND LOAN 
ENTITLEMENTS 


Overview of Loan Application Processing 


The Ministry manages and reports OSAP activity on an operating year of August 1 to 
July 31. Unless otherwise specified, all figures in this report refer to this OSAP operating 
year. As of March 31, 2003, the province had guarantees outstanding on about 594,000 
Ontario Student Loans totalling about $2.4 billion that were held by banks and the 
Ontario Student Loan Trust. About $1.1 billion (47%) was related to borrowers who had 
completed their studies and were repaying their loans. The remaining loans were held by 
students who were still in school or who had completed their studies within the six-month 
period ending March 31, 2003. 


Between August 1, 2002 and March 31, 2003, the Ministry received 204,432 applications 


for financial assistance and approved the amounts outlined in the following table. 


Student Loan Activity August 1, 2002 to March 31, 2003 





Ontario Student Loans 130,824 


Canad 


Source of data: Ministry of Training, Colleges and Universities 


All OSAP-designated schools have a Financial Aid Office (FAO) that serves the school’s 
students. Students who attend Ontario colleges and universities, as well as those who attend 
approximately 50% of the province’s private career colleges, are required to submit all 
supporting documentation to their school’s FAO. (Students attending the remaining private 
career colleges must submit their documentation directly to the Ministry.) The FAQs also 
adjudicate any reviews related to a specific student’s status or request for additional loan 
assistance. All these duties are governed by legislation, ministry policies, and procedural 


guidelines. 


During the 2002/03 academic year, approximately 90% of the applications the Ministry 
received were submitted electronically using the Ministry’s on-line Web site. Applications 
are electronically checked for accuracy, completeness, and eligibility criteria such as Ontario 
residency, registration at an approved postsecondary school in an approved program with at 


least 60% of the full course load, and whether the applicant is in good standing with OSAP. 
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The applicant’s loan entitlement is determined based on such criteria as the student’s family 
circumstances; the reported income of the student and of the student’s parent(s) or spouse 
as applicable; expected educational costs; and the approved living allowances. The federal 
and provincial loan programs are very similar; however, there are a few eligibility criteria that 
differ. Thus, a separate determination of loan entitlement is required for each program. 


After eligibility is approved and the entitlement determined, students must complete a 
“Confirmation of Enrollment” document at their school’s FAO in order for the FAO to 
release their loan certificate. Students forward that “Certificate of Loan Approval and 
Entitlement” to the appropriate service provider. The service provider then either deposits 
funds in the student’s bank account electronically or mails the student a cheque. Students 
normally receive 60% of their loan at the beginning of their first study term and 40% of 
their loan at the beginning of their second study term. 


A student’s assistance entitlements may change if the Ministry subsequently determines that 
the student’s income/financial status, study period, or academic status differs for any reason 
from the information appearing on the student’s application. A reassessment can be made at 
any time. Students are required to provide complete and true information and to notify the 
school’s FAO promptly of any changes to their status. 


Verifying Income 


Students’ income—and, where applicable, their parents’ or spouse’s income—is a key 
criterion for determining loan eligibility and loan entitlement amount. Since 1996 the 
Ministry has attempted to verify incomes reported on each OSAP application, both the 
student’s and any supporting individual’s, by comparing those incomes with incomes 
reported for tax purposes to the Canada Customs and Revenue Agency (CCRA). The 
Ministry verifies reported student, parental, and spousal income with CCRA twice a year, 
usually in November and the following May. If there is a discrepancy between the income 
reported to the Ministry on the OSAP application and the income reported to CCRA, the 
Ministry may reassess the student’s loan entitlement up or down. A student who has 
significantly under-reported income is not eligible for any further loan assistance. 


In 2001 the Ministry began summarizing the results of parental income verification. For 
parental incomes that were verified in November 2001 and 2002, the Ministry determined 
that it had provided excessive loan funding or loan overpayments of $8.6 million to 15,000 
students in these two years. Also, 2,800 of these students were denied their second loan 
disbursements, totalling $3.7 million. 


Ministry policy allows students to carry one loan overpayment regardless of the amount 
overpaid. The amount remains outstanding until students finish their schooling, at which 
point the amount becomes part of their outstanding loan balance. Students begin repaying 
their loans six months after completing their schooling. Students who incur a second 
overpayment are ineligible for further assistance until the amount is repaid. 
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While the Ministry did not routinely summarize the financial impact of spousal and student 
income verification, it did determine from the November 2002 verification that for 
approximately 4,300 students their verified income exceeded the income they reported on 
their application so significantly that it resulted in their permanent restriction from future 
OSAP funding. For a further 28,000 students, the difference was more than $2,000, but 
the total amount of resulting loan overpayments was not determined by the Ministry. 


The Ministry did not analyze the results of income verification to determine the risk of 
undetected abuse. For example, it had not determined the number of parents, spouses, and 
students whose income could not be verified and whether that number was growing or 
declining each year. The Ministry did not know whether there are students who have 
received loans for several years without ever having their or their supporting individual's 
income verified. Having such information would allow the Ministry to more effectively 
assess the risk of abuse over time and to focus its corrective action on areas of particular 
concern. 


Our own analysis of the Ministry’s parental income verification with CCRA for the three 
years ending July 31, 2002 determined that the Ministry was unable to verify parental 
income for more than 15,000 students, or about 7% of dependent students who received 
OSL entitlements during that period. About 9,700 (65%) of these students may have 
qualified for the maximum loan entitlement based on the low parental incomes reported on 
their OSAP applications. We also determined that more than 28,000 students had received 
OSL entitlements without having their own income verified during the three years ending 
July 31, 2001. Of those students, 24,000 (85%) reported incomes below the ministry- 
determined basic cost-of-living threshold of $6,200. These cases warrant investigation given 
the overpayments identified by the Ministry’s parental income verification efforts to date. 


We found that the Ministry was verifying parental income with CCRA on a timely basis. 
However, spousal income was not verified until November of the year following the 
student’s OSAP application—that is, after most students have received three loan 
disbursements. The delay in verification occurs because the Ministry follows the federal 
policy of using spouses’ current year income estimates rather than prior year actual income 
when determining entitlements to both CSL and OSL. These estimates’ accuracy cannot be 
verified until the year after the application is processed. Because they are supporting 
individuals, parents’ and spouses’ incomes have a similar impact on loan entitlement (unless 
the spouse is also a student). Consequently, parental and spousal incomes ought to be 
assessed and verified the same way and at the same time. 


Student income was also not verified until the November following the year in which the 
student was approved for OSAP. Students thus received three loan disbursements before 
their income was verified. More timely verification will depend largely on getting more 
timely data from CCRA. We understand that the Ministry was exploring options with 


CCRA to improve the timeliness of student income verification. 
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Recommendation 


To ensure that efforts to verify incomes reported on applications are effective 
and timely, the Ministry should: 


e analyze the results of income verification to ascertain trends, identify cases 
where it is not working effectively, and take any necessary corrective 
action; and : 

° negotiate policy changes that will permit parental and spousal incomes to 
be verified at the same time. 





Ministry Response 


The Ministry will analyze the results of income verification to ascertain trends 
and identify cases where the process is not working effectively so that 
corrective action can be taken. 


The Ministry is working with the Canada Customs and Revenue Agency to 
improve the timeliness of the income verification process and will explore 
verifying parental and spousal incomes at the same time. 





Calculating Entitlements Where Applicants Report 
Income from Social Assistance 


Both students and the spouses of married students are expected to work during the pre- 
study period to contribute to the student’s educational costs. The Ministry expects a 
minimum contribution no matter how low a student’s actual income is, except where social 
assistance is the main source of that income. 


For students who indicate on their OSAP application that social assistance (Ontario Works, 
Family Benefits, and so on) was the main source of their or their spouse’s reported income 
before starting their studies, a// reported income is excluded from this calculation and the 
minimum contribution is waived, regardless of the total income reported. However, this 
policy can result in students’ receiving more OSAP assistance than they are actually eligible 
for. 


For example, we found several cases in which married students indicated that their spouse's 
income exceeded $20,000, which is the maximum Ontario Works benefit. In four cases, 
the incomes exceeded $50,000, and we referred these cases to the Ministry for follow-up. 
But because these students indicated on their OSAP application that Ontario Works was 
their spouse’s main income source, all other reported income was excluded from the 
minimum contribution calculation. No reasonableness checks were built into the system to 
recognize higher levels of income where Ontario Works benefits or other forms of social 
assistance had been received. 
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Recommendation 





To properly determine entitlements in cases where spousal income includes 
social assistance as a source of income, the Ministry should correct existing 
loan-application processing procedures so that other significant reported 
income sources are considered. 


Ministry Response 


The Ministry will insert additional edit checks for 2004/05 for applications 
where spousal incomes include social assistance benefits. 








REDUCING STUDENT LOAN OVERPAYMENTS 


In addition to instances where income verification has identified that actual incomes are 
higher than students report on their application, loan overpayments can result from other 
changes to a student’s status. The student's academic status or study period can change, or 
the student can formally withdraw. Such changes should normally prompt a reassessment of 
the student’s entitlement. When a student’s OSAP application is reassessed and it is 
determined that the student received more funding than the student was eligible for, the 
student is considered to have received an overpayment. In many cases, the reassessment is 
made after all loan disbursements for the year have been made. Over the four-year period 
from 1998/99 to 2001/02, the Ministry made OSL overpayments to students totalling 
$165 million, as outlined in the following table. 


Amount of Loan Overpayments, 1998/99-2001/02 
70 4 
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($ million) 
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207 
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1998/99 1999/2000 2000/01 2001/02 


: 1999/2000 was the first year that the Ministry began adjusting entitlements due to under-reporting 
of student income, contributing to the significant increase in identified overpayments. 


Source of data: Ministry of Training, Colleges and Universities 
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As stated earlier, students are allowed to carry one overpayment until they finish their 
schooling. As long as the student remains in school, the Ministry pays on the student's behalf 
the interest that accrues on that student’s loan. Thus, taxpayers are subsidizing the interest 
carrying costs on any overpayments. At an average interest rate of 5.8%, these costs 
amounted to approximately $2.5 million annually between 1998/99 and 2001/02. In 
addition, the taxpayers reimburse the banks and the Ontario Student Loan Trust for all 
defaulted loans, including any overpayments. Based on the 2002 default rate of 13.9%, 
overpayments between 1998/99 and 2001/02 have likely cost taxpayers at least $6 million 
annually (and likely more, because earlier years’ default rates were higher). By minimizing 
overpayments, the Ministry would reduce interest costs and also reduce the amount of loans 
at risk for future loan defaults. 


The Ministry does not analyze the main sources of overpayments to determine needed 
corrective action. Two major reasons why loan overpayments occur are under-reporting of 
either the student’s own or a supporting individual’s income, and early withdrawal from 
studies. The Ministry could take at least three actions to reduce overpayments, as discussed 
below. 


Adjusting the Timing of Loan Advances 


One way to reduce the risk of significant overpayments and the associated interest and 
default costs is to match assistance payments more closely to students’ cash flow 
requirements. Some jurisdictions advance student loan funding on a monthly basis. 
However, Ontario students who are enrolled in three terms, or 52 weeks, of study, often 
receive all of their loan disbursements after attending only one term of study. These students 
still have 34 weeks of studies to complete. 


Our analysis of loans approved in 2001/02 determined that some 8,600 students of the 
19,000 students enrolled in three terms of study that year had received their entire loan by 
early in the second term. For example, one student’s loan entitlement was $20,460 to attend 
a 51-week program from September 2001 to August 2002. The entire amount had been 
paid to the student by the beginning of January 2002, even though the student still had 


eight months of classes to complete. 


Advancing all of the loan funding before students have completed the majority of their 
studies creates the potential for large overpayments if a student's loan entitlement changes 
(for example, due to a change in the student’s financial or academic status). The timing of 
loan disbursements could thus be a significant contributing factor to the number and size of 
loan overpayments. In addition, better matching of loan advances to students’ cash flow 
requirements may assist students to prudently manage their finances. 


Monitoring Academic Status 


OSAP recipients are required to maintain full-time status (defined as taking at least 60% of 
a full course load) in order to remain eligible for OSL assistance and any OSOG. The 
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Ministry requires Financial Aid Offices (FAQs) to monitor academic status during the 
school year and to report students who withdraw from school or reduce their course load to 
part-time status. 


During our review of student files at the FAOs.we visited, we found several students whose 
status should have been reported as part-time but who were recorded in the OSAP system 
as full-time. After our visits, the FAO staff corrected the students’ status. These status 
changes reduced loan entitlements, thereby creating loan overpayments on which the 
Ministry continues to pay interest. 


As stated earlier, one factor contributing to loan overpayments is a student’s early withdrawal 
from studies. A reported withdrawal triggers a loan entitlement revision. Timely reporting of 
student withdrawals is thus critical for limiting loan overpayments. The Ministry indicated 
to us that it expected FAOs to report withdrawals to the OSAP system within 30 days, 
although ministry policy was not clear in this regard. We found, however, that for 2000/01 
and 2001/02 about 60% of the withdrawals reported by postsecondary schools to the 
Ministry (6,300 students) were reported more than 30 days after the actual withdrawal 
date. 


Thus, both advancing the majority of loan funding early in the study period and reporting 


withdrawals late increase the risk of significant loan overpayments. 


Reviewing Requirements for Repayment 


The Ministry might also consider changing policies that may inhibit prompt and accurate 
reporting by students. As stated earlier, the Ministry allows students to carry one loan 
overpayment until they finish school, regardless of the amount overpaid. However, loan 
overpayments always result from the student’s own actions, whether deliberate or 
inadvertent. If assistance is to be fairly administered, both the cause and amount of the 
overpayment ought to be a factor in deciding what action to take. However, only in cases 
where significant under-reporting of income is detected will the student be denied future 
assistance and be required to begin repaying the loan. 


We determined that in 2000/01 and 2001/02, approximately 18,000 students received an 
OSL overpayment of between $1,000 and $5,000. A further 554 students in these two 
years received an OSL overpayment that exceeded $5,000. We calculated that the interest 
paid by the Ministry on overpayments greater than $1,000 was approximately $1.7 million 
and $550,000 for these two years, respectively. This is a significant portion of the 

$2.5 million we estimate the Ministry pays annually on overpayments. Limiting the amount 
of allowable overpayment, particularly in cases where the student has not reported accurate 
information to the Ministry, would therefore save significant interest costs and discourage 
program abuse. 
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Recommendation 


To minimize the occurrence and size of loan overpayments and to reduce the 
related interest and default costs, the Ministry should: 


e analyze loan overpayments yearly to determine the main reasons for them 
and take corrective action based on the results; 


e match the timing of loan disbursements more closely to students’ cash flow 
requirements; 


e ensure that all Financial Aid Offices monitor students’ academic status 
during the entire year and promptly record any changes to that status on 
the OSAP information system in accordance with ministry expectations; 
and 


e consider limiting the amount of loan overpayment that a student who has 
not reported accurate information to the Ministry is permitted to retain. 


Ministry Response 


The Ministry will undertake analyses of the reasons for loan overpayments and 
consider changing the number of disbursements from two to three for 
students enrolling for three terms in a year, possibly beginning in 2004/05. 


The Ministry will work with Financial Aid Offices at institutions to provide more 
timely reporting of changes to academic status. 


The Ministry’s loan overpayment policy attempts to balance students’ needs for 
funding to complete their studies against additional costs to taxpayers. The 
current policy allows students to defer one loan overpayment while they are in 
school and to repay it after they have completed their studies and are in a 
better position to repay the loan. 














CONTROLLING ONTARIO STUDENT 
OPPORTUNITY GRANT PAYMENTS 


The Ontario Student Opportunity Grant (OSOG) replaced the Loan Forgiveness Program 
in 1998/99. OSOGs help students reduce their annual student debt level to $7,000 for a 
two-term academic year or $10,500 for a three-term academic year. Students must meet 
certain other requirements before receiving an OSOG. The main requirements are 
completing their registered terms (whether two or three) on a full-time basis, having income 
verified with CCRA (as we recommended in 1997 for the previous Loan Forgiveness 
Program), and not having defaulted on any previous student loans. 


Students do not have to apply for an OSOG. Once the eligibility requirements have been 
met, the OSOG is paid directly to the service provider to pay down the student’s loan. Over 
the 2001/02 and 2002/03 fiscal years, the Ministry paid out over $241 million in OSOGs. 
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We visited several postsecondary schools and examined the academic records of students 
who received OSOGs to determine whether they had met the Ministry requirements. We 
also tested a sample of student files from other schools across the province. We concluded 
that most OSOG payments were being calculated correctly and were being paid only to 
eligible students. However, we found that OSOG payments had been made incorrectly in 
three situations: where students were deemed to be full-time but were actually part-time, 
where three-term students who received an OSOG did not complete their third term on a 
full-time basis; and where students had failed to complete their academic year but had not 


formally withdrawn. 


We found three cases in our sample where the student was taking only a part-time course 
load. Because they were never reclassified from full-time to part-time status, these students 
received about $17,000 in OSOGs they were not entitled to. We found a further six 
students in our sample who had completed only two terms as full-time students but were 
granted OSOGs for three terms. These students, too, incorrectly received OSOGs totalling 
approximately $17,000. Although the FAOs involved corrected the OSAP records once we 
brought the cases to their attention, the Ministry had no process for recovering the 
incorrectly paid OSOGs from the Ontario Student Loan Trust. 


The OSOG regulation states that if a student withdraws or is expelled from one or more 
academic terms, any loan received in that term is ineligible for an OSOG. We reviewed a 
sample of student academic records at the schools we visited to determine if students had 
genuinely attempted to complete the academic year. To qualify for an OSOG, a student 
does not need to succeed academically, but the student should at least be clearly trying to 
complete his or her studies. However, we saw a risk that certain students might not formally 
withdraw but might nevertheless make no attempt to complete the year by attending classes, 
submitting assignments, and/or writing final examinations. 


To determine whether this was occurring, we reviewed a sample of OSOG recipients’ final 
grades for 2001/02 for evidence that students had actually attempted to complete their year 
rather than simply quitting but not formally withdrawing. We did find seven students 
whose grades were all zeros or Fs. These students may have incorrectly received up to 
$12,000 in OSOGs. We believe it is quite possible that such students did not genuinely try 
to complete their year, but the FAQs involved did not have a process to identify and 
investigate such students. 


Due to the nature of postsecondary education (particularly the lack of formal attendance 
requirements), it is difficult for FAOs and the Ministry to verify a student’s academic status. 
However, we found that midway through the year, before the second loan disbursement, a 
few FAOs were trying to identify students who had stopped attending classes. These FAOs 
first identified all students with low grade-point averages, and then required those students 
to submit a form signed by their professors confirming that the student had attended classes 
regularly. Students who did not submit properly signed forms no longer qualified for OSAP, 
and their second loan disbursement was therefore not paid. However, most FAQs did not 


make this kind of effort. 
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To produce similar savings for OSOGs, the Ministry would need to require all schools to 
establish a similar process at the end of the year (in conjunction with the academic progress 
evaluations that already take place). Determining at that point which students had stopped 
attending classes before the end of the term would prevent the Ministry from paying 
OSOGs on behalf of students who are not legally entitled to them. 













Recommendation 


To ensure that only eligible students receive Ontario Student Opportunity 
Grants, the Ministry should work with postsecondary schools to identify 
students who reduce their course load to part-time status and students who 
do not formally withdraw from their program but make no attempt to complete 
the academic year. 


Ministry Response 





The Ministry will work with Financial Aid Offices at institutions to identify and 
implement best practices that will lead to more timely and accurate reporting 
of students’ academic status. 


MANAGING THE RISK OF PROGRAM ABUSE 


Many components of OSAP are administered in a decentralized manner, relying on 
postsecondary schools to play a key role in ensuring that OSAP responds to student needs 
and that legislated and OSAP policy requirements are met. 


In response to our 1997 audit, the Ministry established specific performance requirements 
in order to better manage its relationships with the schools. In addition, the Ministry 
implemented OSAP compliance audits for all three types of schools. The objective of the 
audits is to provide the Ministry with independent assurance that FAOs are administering 
OSAP in accordance with ministry policies and procedures. 


Compliance audits for all OSAP-eligible private career colleges have been completed 
annually since 1998. Audit results have steadily improved each year. In addition, special 
audits were conducted in the few cases where serious deficiencies were detected by previous 
compliance audits. The Ministry also requires private career colleges to publish graduation 
and employment rates each year for each of their OSAP-approved programs. 


Publicly funded colleges and universities were last audited in OSAP’s 1998/99 fiscal year; 
the next round of audits was scheduled to be completed in mid-2003 for OSAP’s 2001/02 


fiscal year. The initial audits did not detect significant instances of non-compliance. 


While the Ministry has introduced several initiatives to improve administration and reduce 
the risk of program abuse by schools, it has done less to identify risks and incidences of abuse 
by students. The Ministry already captures extensive information on students that can be 
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used or analyzed to reduce the risk of abuse. But the Ministry had not determined the types 
of analyses that would be most effective in identifying students who abuse OSAP. 


We undertook several such analyses as part of our audit. For example, we detected a 
significant number of sole-support and married students who received additional funding 
by claiming an increase of three or more children from the previous year. Specifically, we 
found 78 students who reported an increase of three or more dependent children from the 
previous year: 21 of these students reported an increase of five or more dependent children. 
We advised the Ministry of this, and subsequent investigation by the FAOs involved when 
following up on our request for the documentation determined that most of these 
applications had been completed incorrectly and that the errors were likely inadvertent. All 
errors were promptly corrected. In 20% of the cases we examined, the student’s loan 
entitlement had to be reduced as a result of this correction, creating a loan overpayment. 
Moreover, the risk that this type of error will go undetected is greater for married 
students—because, unlike sole-support students, they are not required by the Ministry to 
submit documentation to support the number of children they claim. 


We also found 181 parents of OSAP-eligible students who reported an increase of three or 
more dependent children in one year. Of these, 46 reported an increase of five or more 
dependent children. Again, the student’s loan entitlement was likely inflated in several of 
these cases. We referred these cases to the Ministry for follow-up. 


The risk of OSAP abuse can be high. The Ministry stated in its 2001/02 business plan that 
it was planning to establish consistent baseline data on the incidence of fraud and 
inappropriate use of OSAP. Also, the Ministry was planning to establish a hotline for 
individuals to report suspected OSAP abuse. Neither plan had been implemented at the 


time of our audit. 


The Ministry’s OSAP Inspection and Compliance Unit’s Student Verification Officers are 
responsible for examining student loan files to verify information provided, determine 
compliance with OSAP criteria, and undertake initiatives to reduce OSAP abuse. The 
Ministry did not maintain a team of student verification officers during the 2002/03 year. 
As a result, the Ministry did not track the source, frequency, nature, and results of the cases 
that were referred to the unit for review. 


Alberta has a similar group that verifies or audits student loan accounts. The audits or 
verification efforts are categorized according to what triggers them—for example, public 
complaints, program staff referrals, or data matching with other government information 
sources such as social assistance or CCRA records. They also track their audit results by the 
frequency and amount of over-awards identified to detect chronic problem areas and to 
demonstrate the cost-effectiveness of the audit process. 
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Recommendation 


To minimize the risk of OSAP abuse by students, the Ministry should use its 
extensive database to identify individual cases of potential abuse and analyze 
summary statistics for possible trends warranting investigation and, where 
necessary, appropriate corrective action. 


Ministry Response 


The Ministry will consider expanding its use of the OSAP database to identify 
individual cases of potential abuse by students and to identify trends 
warranting investigation. 





REDUCING DEFAULTED STUDENT LOANS 


Students are expected to repay their loans, including any overpayments. But the province 
guarantees all student debts, so the Ministry reimburses the financial institution or the 
Ontario Student Loan Trust for any defaulted student loans. The Ministry then sends 
default claims to private collection agencies, which attempt to recover these funds from the 
borrowers on the government's behalf. 


Since 1997/98, the Ministry has paid approximately $783 million to the financial 
institutions to cover 143,300 defaulted student loans. Although they have been generally 
declining, as indicated in the following chart, defaults on student loans are a major OSAP 
cost—in 2002/03, they represented nearly 27% of OSAP expenditures. 


Student Loan Defaults, 1997/98—2002/03 
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Because of the escalating default trend at the time of our 1997 audit, the Ministry took 
several actions, including: publishing annual default rates by school and program; removing 
OSAP approval from schools that were not meeting performance requirements; requiring 
schools with high default rates to implement default reduction strategies and to share 
default costs; and introducing credit screening for new loan applicants. These initiatives 
undoubtedly contributed to a decline in the default rate, from 23.5% for 1997 to 13.9% 
for 2002. 


The Ministry’s goal is to reduce the default rate to less than 10% by 2005. By comparison, 
the loan default rate in the United States (where the same methodology is used for 
calculating default rates) for the 10-year period ending in 2002 declined from 22% to 
5.9% due to many changes in loan management and the implementation of various default 
management strategies. 


In Ontario, based on the average 2001/02 default loan claim, every 1% reduction in the 
default rate saves at least $3.5 million annually. If the Ministry can reach the same default 
rate as the United States, the province would save a further $28 million annually. 


Adopting Practices of Other Jurisdictions 


As noted above, the Ministry has implemented many tools and strategies to reduce its 
default rate, but additional measures may be needed to ensure that the default rate 
continues to decline. Other jurisdictions have introduced several measures aimed at 
lowering their default rates and the cost of defaulted loans. For example: 


¢ The Ministry could remind students each year that they have a loan they must repay 
when they leave school by providing them with an annual statement showing how 
much they have borrowed to date. 


¢ The Ministry could also identify high-risk borrowers, such as married and sole-support 
students, who would benefit from special counselling, either individually or in groups, 
to help them successfully establish and complete repayment and avoid defaults. Between 
1998/99 and 2001/02, married and sole-support students accounted for less than 20% 
of total loans issued, but 40% of total loan defaults. As weil, the current contract with 
the service providers does not require them to focus on these high-risk cases. 


¢ Students who leave school before completing their programs, especially within the first 
year of enrolment, are known to have a high likelihood of defaulting on their loans. For 
2000/01 and 2001/02, the default rates for students who did not complete their 
studies were more than double the default rates for students who completed their 
studies. We understand that the federal government is considering implementation of 
an early withdrawal management program. 


As part of our audit, we conducted a survey of college and university FAOs to identify best 
practices for encouraging loan repayment and to determine areas of needed improvement. 
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In addition to the approaches noted above, many of the FAOs that responded made the 


following suggestions: 


¢ Mandatory entrance counselling on OSAP responsibilities could be required before 
students receive their loans. This practice has been used in the United States for several 
years. FAQOs also indicated that effective exit counselling once a student’s education was 
nearly complete was important because students often lack information, experience, and 
assistance in managing their finances and meeting their debt responsibilities. 


¢ Better communication with students could take place to educate them about their debt 
responsibilities and about the available repayment options. Many FAQs stated that if 
this information was provided at the time of loan disbursement or at the start of each 
semester, students would be more aware of their current level of indebtedness. 


Use of the Income Tax Set-off Program 


In January 1999, to improve collections of defaulted student loans, the Ministry 
implemented an income tax set-off (ITSO) program where efforts to collect defaulted 
student loans had been unsuccessful. Under the ITSO program, the Governments of 
Canada and Ontario can withhold a defaulter’s income tax refund and apply the proceeds 
against the amount owing on a defaulted student loan. As stated earlier, all defaulted 
Ontario Student Loans are first sent to private collection agencies, which attempt to recover 
the loan amount. After the private collection agency has attempted to collect funds for a 
minimum of one year without success, the Ministry may ask that the account be returned to 


the Ministry and set up for ITSO. 


The Canada Customs and Revenue Agency is required to notify people identified for ITSO 
that their income tax refund may be withheld. Defaulters then have the opportunity to 
make arrangements with the Ministry to start repaying the outstanding debt instead of 
having their income tax refund withheld. Up to March 31, 2003, the Ministry had 
collected $8.4 million from this initiative: $6.8 million from ITSO and $1.6 million from 
people who had made payment arrangements with the Ministry to avoid ITSO. 


Since the ITSO program started, the Ministry has asked that approximately 27,000 
accounts be returned from the private collection agencies for TSO. However, from 
1996/97 to 2000/01, approximately 120,000 defaulted accounts would have met the 
Ministry’s initial eligibility criteria of at least one year of collection activity. Ministry staff 
informed us that staffing constraints limited how many defaulted student loan accounts 
could be set up for ITSO: the Ministry was not equipped to handle a large volume of 
debtor inquiries related to ITSO accounts. In January 2003, the Ministry increased the 
number of accounts established for [TSO by 40% from the previous year, but there are still 
at least 60,000 accounts available that could be set up for ITSO. 
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Recommendation 


To continue reducing the losses arising from defaulted student loans, the 
Ministry should: 


e implement best practices used successfully in other jurisdictions to reduce 
the risk and cost of defaulted student loans; and 


e establish income tax set-off arrangements for all defaulted loan accounts 
for which normal collection efforts have been exhausted. 





Ministry Response 


Within Canada, Ontario is a leader in default prevention. The Ministry expects 
further decreases in the default rate as a result of new default management 
arrangements in place or planned with the federal government and its service 
providers. For example: 


e the service providers can work for up to 270 days with borrowers who are 
experiencing difficulties making their loan payments, whereas banks 
declare loans in default after just 90 days of non-payment; 

¢ Ontario and the federal government have introduced extended interest 
relief and debt remission programs to provide more help to borrowers who 
are experiencing difficulties; 

e students will soon be able to go on-line to check their accounts; and 

¢ the service providers are developing a risk model to identify students who 
may be more prone to defaulting. 


The Ministry is seeking to make arrangements that would allow it to move a 
larger number of defaulted loan accounts to the income tax set-off program. 
The Ministry will approach Management Board’s Collection Management Unit 








to explore the possibility of undertaking an aging analysis of defaulted loans in 
collections and to review the criteria for transferring defaulted loans to the 
income tax set-off program. 





EXPANDING PERFORMANCE REPORTING 


In our 1999 follow-up of recommendations made regarding OSAP in the 1997 Annual 
Report, we reported that the Ministry had introduced several performance measures. We 
also reported that plans for harmonizing the federal and provincial assistance programs 
would allow the Ministry to compare these performance measures, as well as new measures 
on student indebtedness and ability to pay, with parallel measures in other jurisdictions. 


In May 1999, the Canada-Ontario Agreement on Harmonization of Federal and Provincial 


Student Loans Programs was signed. The measures agreed to at that time were: 


¢ Access—the ability of individuals to pursue postsecondary studies regardless of the level 
of financial support available from their families; 
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¢ Completion/non-completion rates—the proportion of students completing their 
program of study; 


¢ Student indebtedness and capacity to pay—the amount of payment relative to the 
individual’s income; and 


¢ Default rates. 


In July 2001, both parties signed an integration agreement for student loans. The 
performance measures previously agreed upon remained unchanged. 


In 2001/02, the Ministry’s only published performance measure for OSAP was default 
rates program-wide, by school, and by individual program of study. The Ministry also 
reported graduation and employment rates for each OSAP-approved private career college 
program. (Public postsecondary schools publish graduation and employment rates for each 
of their programs themselves.) 


In April 2003, the federal government and most provinces agreed to a national 
“Designation Framework.” (Designation is the process used to approve institutions for 
student loan purposes.) The Framework’s four principles are taxpayer protection, 
accountability and informed choice, consumer protection, and complementarity to other 
postsecondary education policies. 


According to the Ministry, this Framework is now the basis for the development of OSAP 
performance reporting. All participating provinces are to develop and report common 
performance measures to allow comparability. The three categories of performance 
measures agreed upon by all jurisdictions were: 


¢ Portfolio performance—for example, repayment data and default data; 


¢ Institution performance—for example, administrative compliance and student support 
services; and 


¢ Student performance—for example, completion data, employment data, and 


withdrawal data. 


While some of the proposed measures are available from the Ministry’s current performance 
information, other existing measures may have to be amended and some new measures may 
have to be introduced. 





Recommendation 


To evaluate and report on the effectiveness of Ontario Student Assistance 
Program and strengthen accountability, the Ministry should establish deadlines 
to begin publicly reporting the agreed-upon performance measures. 
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Ministry Response 


As the Provincial Auditor states, Ontario already has introduced public 
reporting of performance measures, including graduation and graduate- 
employment rates and student loan default rates. Federal, provincial, and 
territorial officials are currently working to develop and finalize the 
performance measures. Once finalized, the measures will be combined to 
create an institutional risk “scorecard” to identify where action with a 
particular institution is required. Officials are required to report back to 
Ministers on progress at the Council of Ministers of Education, Canada 
meeting in fall 2003. Jurisdictions are expected to have developed their 
performance measures by September 2004. Implementation is scheduled to 
begin in fall 2004. 
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CHAPTER FOUR 


Follow-up of 
Recommendations in the 
2001 Annual Report 





It is our practice to make specific recommendations in our value-for-money (VFM) audit 
reports and ask ministries and agencies to provide a written commitment to take corrective 
action for publication in Chapter Three of our Annual Report. Two years after we publish 
the recommendations and related responses, we follow up on the status of actions taken by 
ministries and agencies with respect to our recommendations. 


Chapter Four provides some background on the value-for-money audits reported on in 
Chapter Three of our 2001 Annual Report and describes the current status of corrective 
action that has been taken to address our recommendations since that time. The sections in 
this chapter consist of follow-up reviews. 


Our follow-up reviews are planned and conducted to provide a moderate level of assurance 
on the extent and adequacy of corrective actions taken by ministries or agencies. The reviews 
consist primarily of inquiries and discussions with management; analyses of information they 
provide; and, where deemed appropriate, limited examination and testing of systems, 
procedures, and transactions. This is not an audit, and accordingly, we cannot provide a 
high level of assurance that the corrective actions described have been implemented 
effectively to fully resolve noted problems. Where these reviews identify new issues or 
concerns that need to be acted upon, these issues will be considered in our audit planning 
process. In any event, the results of the corrective actions taken or planned will be more fully 
examined and reported on through future audits. 


Overall, while a number of our 2001 recommendations appear to have been substantially 
acted upon, there are still many recommendations where action remains to be taken or is 
ongoing. 
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MINISTRY OF AGRICULTURE AND FOOD 


4.01—Food Industry Program 
(Follow-up to VFM Section 3.01, 2001 Annual Report) 





BACKGROUND 


The objective of the Food Industry Program of the Ministry of Agriculture and Food (at the 
time of our audit named the Ministry of Agriculture, Food and Rural Affairs) is to manage 
food safety risk in Ontario’s food industry to protect consumers and enhance market access 
and industry competitiveness. In 2000, Ontario’s food industry included over 60,000 farms 
that produced $7.8 billion worth of agricultural production and food processors that 
shipped products to market valued at $25 billion. 


‘To maintain the safety and quality of the province’s food supply, the Ministry, in 
co-ordination with other provincial ministries as well as the federal and municipal 
governments, licenses and inspects food processing plants and tests selected products for 
evidence of contamination. To support the program, for the 2002/03 fiscal year, the Food 
Industry Division spent a total of $20 million. In 2000/01, it spent $20 million, employed 
110 staff, and engaged 140 inspectors on a contract basis. 


Although the Ministry had engaged in a number of initiatives to protect consumers from 
food-borne contaminants, we concluded that the Ministry needed to improve its efforts to 
ensure compliance with legislation, policies, and procedures by addressing weaknesses in its 
licensing and inspection processes. Specifically, we noted the following: 


¢ Food safety deficiencies that were defined as critical by the Ministry and that can pose 
risks to human health were noted during annual licensing audits of abattoirs 
(slaughterhouses) but often were not corrected in a timely manner. 


¢ Newer testing methods were not in place to randomly test meat from abattoirs for 
evidence of bacterial, chemical, and other more recently recognized hazards to health 
that are not readily detected by traditional meat inspection methods. 


¢ The Ministry had not assessed the effectiveness of the Dairy Farmers of Ontario (DFO) 
since 1998 when the DFO had assumed responsibility for ensuring the safety of the 
processing of raw cows’ milk. Consequently, we reviewed the activities of the DFO and 
found that adequate inspection processes for raw (unpasteurized) cows’ milk had been 


established. 


¢ One-third of the goat dairy farm inspection reports we reviewed were given a 
conditional rating because of non-compliance with minimum standards, and 90% of 
the goats’ milk samples tested on behalf of the Ministry did not meet legislated bacterial 
standards. There was no evidence that the Ministry had followed up on these problems. 
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¢ In 2000, the Ministry tested almost 800 fruit and vegetable samples and found 28 cases 
where chemicals exceeded acceptable limits by as much as 80 times the maximum 
allowable limit. As of March 2001, the Ministry had yet to formally notify growers and 


retailers of the test results. 


¢ Penalties imposed for non-compliance with food safety legislation were not sufficient to 
have a deterrent effect. During the 2000/01 fiscal year, the average penalty was $320 
for infractions such as illegally slaughtering poultry and illegally processing cheese. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


According to information received from the Ministry, substantial action has been taken on 
all the recommendations made in our 2001 Annual Report. Some of the action taken by the 
Ministry will be addressed through the Food Safety and Quality Act, 2001 and associated 
regulations. This Act received royal assent on December 5, 2001 but had not been 
proclaimed as of May 2003. Therefore, the requirements of the Act were not in force at the 
time of our follow-up review. The current status of each of our recommendations is outlined 
below. 


COMPLIANCE WITH LEGISLATION, POLICIES, AND 
PROCEDURES 


Abattoir Licensing, Inspection, and Testing—Licensing 


Recommendation 
To better ensure that all provincially licensed abattoirs meet licensing requirements and the meat 


produced is safe for human consumption, the Ministry should: 


¢ develop specific criteria for licence suspensions and other penalties for non-compliant 
abattoirs; 


° establish guidelines for setting acceptable time frames for abattoirs to take corrective 
action on problems noted during the licensing audits; and 


° ensure abattoirs take corrective action as soon as possible after licensing audits are 


completed so that food safety issues can be resolved promptly. 


Current Status 


The Ministry informed us that it developed specific criteria for licence suspensions and 
other penalties, and introduced an enhanced rating system on April 1, 2002 that better 
reflects the food safety status of provincially licenced abattoirs. The new system is based on a 
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letter grade and is similar to that used by the Canadian Food Inspection Agency in federally 
inspected plants. The new system allows the Ministry to ensure non-compliant operators 
take timely corrective action in that, after an abattoir is audited, feedback is immediately 
provided. For example, plants receiving an “A” rating or higher are to have a corrective- 
action-plan meeting with the inspector within four weeks of the audit being completed to 
set time frames for correcting major deficiencies. Plants receiving lower ratings would have 
corrective action initiated earlier than four weeks. For 2002/03, the Ministry indicated that 
80% of the 209 provincially licensed abattoirs received a rating of “A” or higher. 


Abattoir Licensing, Inspection, and Testing—Inspections 


Recommendation 


To help ensure that corrective actions arising from annual licensing audits are completed as 
required and food safety risks are minimized, the Ministry should ensure that inspectors follow up 
on and adequately document the completion of all operational tasks. 


Current Status 


The Ministry indicated that, to ensure that inspectors have the necessary tools to adequately 
monitor the required corrective actions resulting from licensing audits, it has provided 
training in the areas of information systems, meat hygiene, and meat processing. Inspectors 
also monitor the abattoirs’ progress in correcting deficiencies through the Food Safety 
Decision Support System that records information from all abattoir audits and inspections. 
In addition, area managers are now responsible for ensuring that inspectors follow up on 
and adequately document the completion of all operational tasks. 


Abattoir Licensing, Inspection, and Testing—Laboratory 
Testing of Meat 


Recommendation 


To help ensure that meat products from provincially licensed abattoirs are safe for human 
consumption, the Ministry should: 


¢ develop a risk-based approach to laboratory testing and utilize newer methods to test 


meat from healthy animals for potential food safety hazards; 
° summarize laboratory results to help identify systemic problems; and 


° ensure that all problems detected are followed up on and resolved in a timely manner. 


Current Status 


The Ministry informed us that it improved its risk-based process for random and targeted 
sampling of animals for chemical residues. It is now testing cows, calves, lambs, goats, 
chickens, turkeys, and sows for antibiotic residues. In addition, testing for high-risk livestock, 
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such as barbeque hogs, has been enhanced to detect sulfonamide residues. The Ministry 
also conducted a trial test at a high-risk cow abattoir using US microbial performance 
standards. As well, baseline data for microbiological and chemical contamination of beef, 
hogs, and chickens have been developed. Results from the trial test should help the Ministry 
to develop a process for ongoing monitoring for microbial contamination to ensure that 
meat products processed in provincially licensed abattoirs meet high standards. 


In addition to carrying out on-site testing at some cattle and hog plants, the Ministry has 
developed new laboratory testing and rapid-testing technology, such as carbadox detection 
methodology, sulfa-on-site tests, and an on-premise swab test for antibiotic residue. 
Laboratory results from this testing are now reviewed monthly by managers to identify 
systemic problems and ensure appropriate and adequate follow-up. The new procedures 
also establish time frames for correcting deficiencies on a timely basis. 


In 2001, the Ministry enhanced its comprehensive testing program for bovine spongiform 
encephalopathy (BSE) (commonly referred to as “mad cow disease”). This ongoing program 
is designed to test the highest-risk animals, including animals that die on the farm, older 
animals, and animals that display signs of neurological distress. During the 2002/03 fiscal 
year, the Ministry informed us that more than 1,100 samples had been tested for BSE in 
Ontario, which is three times the internationally recognized testing standard. All samples 
tested negative for BSE. 


As well, managers now receive monthly summaries of all laboratory results, and the Ministry 
has implemented protocols for follow-up of unfavourable results within established time 
frames. The Ministry informed us that compliance with these protocols is being monitored 
on a regular basis. 


Animal Disposal Industry: Licensing and Inspection 


Recommendation 


To better ensure that deadstock operators comply with legislation and that meat from deadstock is 
properly disposed of and does not end up in the human food chain, the Ministry should: 


° inspect the vehicles of deadstock collectors in accordance with ministry policy to ensure 
that valid permits are in place; 


° implement a formal policy for the frequency of inspections of deadstock collectors, 


rendering facilities, receiving plants, and brokers; 


¢ obtain and review all federal inspection reports and follow up on any areas not covered 
by federal inspectors; and 


° review and revise as necessary legislation, policies, and procedures concerning the 
transportation of dead animals. 
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Current Status 


The Ministry informed us that a study was completed regarding current legislation and 
regulations pertaining to the transportation of dead animals, inspection of vehicles, and the 
licensing of deadstock operators. As a result, the Ministry is developing new regulations 
under the Food Safety and Quality Act, 2001 and the Nutrient Management Act, 2002. \n 
addition, the Ministry has revised its inspection procedures and appointed a co-ordinator to 
ensure inspections of deadstock collectors, receiving plants, and brokers are carried out and 
that noted deficiencies are corrected. Inspections are now scheduled based on a risk 
assessment of the deadstock collectors and their history of compliance with policies and 
timeliness in addressing deficiencies. Receiving plants, brokers, and collectors are inspected 
annually, quarterly, or monthly as determined by the risk assessment. 


The Ministry indicated that, for the 2003 licensing year, it inspected all the deadstock 


collection vehicles and set up a system to ensure that valid vehicle permits are in place. 


The Ministry now receives, within 30 days of the inspection, all federal inspection reports 
for the six rendering facilities that the Canadian Food Inspection Agency inspects. At a 
minimum, rendering facilities are inspected annually. The Ministry reviews these reports 
and follows up on areas not covered by the federal inspectors before renewing the licences 
for these facilities. 


Dairy Licensing, Inspection, and Laboratory Testing— 
Dairy Farms and Milk Transportation 


Recommendation 
To better ensure the quality of raw cows’ and goats’ milk, the Ministry should: 


° periodically review the inspection activities of the Dairy Farmers of Ontario to ensure 
the established standards are being maintained; 


° prepare an up-to-date list of all goats’ milk dairy producers and establish a formal risk- 
based inspection process for goat dairy farms; and 


° ensure that problems identified in inspection reports for goat dairy farms are being 
corrected on a timely basis. 


Current Status 


The Ministry commissioned a thorough review of the Dairy Farmers of Ontario Raw Milk 
Quality Program and found a high degree of compliance with established standards. Some 
recommendations for procedural improvements were made, and these are expected to be 


implemented by fall 2003. 
With respect to goats’ milk, the Ministry indicated that it had completed a list of producer 


and processor information, which is maintained on a computer database. Along with 
developing a new inspection checklist for goats’ milk farms, the Ministry hired two 
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inspectors to further enhance the inspection process. Correction of deficiencies noted in 
inspection reports is now monitored by the Ministry to ensure that goat farmers correct 
these on a timely basis—if they do not, the shipping of milk is now being suspended. In 
February 2002, the Ministry completed a review of the goats’ milk quality program and 
plans to update the regulations (including establishing penalties), improve laboratory testing 
and reporting, provide additional inspection resources, and complete a baseline study of 
goats’ milk in the province. Policy development, consultation, and drafting of new 
regulations under the Food Safety and Quality Act, 2001 is expected to be completed in 
early 2004. 


Dairy Licensing, Inspection, and Laboratory Testing— 
Dairy Plants and Distributors 


Recommendation 


In order to better ensure that all dairy plants and fluid milk distributors comply with relevant 
health and safety requirements, the Ministry should: 


© renew licences on a timely basis for all dairy plants and fluid milk distributors; 
° inspect all dairy plants annually as required by ministry policy; 


¢ follow up on any concerns noted during federal inspections of dairy plants prior to 
renewing licences; 


° ensure that all corrective action plans to remedy deficiencies at dairy plants are 


reviewed and appropriate follow-ups are performed; and 


° review the inspection process for fluid milk distributors and revise its inspection 
requirements accordingly. 


Current Status 


To ensure that licences are renewed on a timely basis for all dairy plants and fluid milk 
distributors, the Ministry is using the Food Safety Decision Support System to improve 
tracking of licence renewal dates and follow-up action required to correct deficiencies. In 
addition, the Ministry hired additional staff and plans to inspect all dairy plants during 
2003. Concerns noted during the federal inspections of dairy plants must now be resolved 
before the Ministry will renew any licences. Any deficiencies noted during either the federal 
or the provincial inspections must be corrected within 30 days, and the Ministry was 
developing procedures to formalize this process to ensure corrective action plans are in 
place. To better co-ordinate the licensing and inspection process for the production, 
processing, and distribution functions, the Ministry consolidated these responsibilities into 
the Dairy Food Safety Unit. Also, a review of the fluid milk distribution program was being 
carried out, and the Ministry was hoping to make any required regulatory changes to the 
program by early 2004. 
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Dairy Licensing, Inspection, and Laboratory Testing— 
Dairy Laboratory Testing 


Recommendation 
To better ensure the safety and quality of Ontario’ cows’ and goats’ milk, the Ministry should: 


° establish bacterial standards for finished milk products and adopt temperature control 
standards for the transportation of raw milk; 


° collect, promptly transport, and test raw milk samples from all goat dairy farms at least 
monthly, as required by legislation; and 


¢ follow up on and ensure corrective action is taken in all instances where goats’ milk 
producers exceed the legislated bacterial standard. 


Current Status 


The Ministry indicated that it was working on establishing bacterial standards for finished 
milk products through changes to regulations under the Milk Act (for cows’ milk) and the 
Food Safety and Quality Act, 2001 (for goats’ milk). In addition, the standards for 
temperature control during the transportation of raw milk and at processing plants that are 


specified in the National Dairy Code will be adopted as part of the proposed regulations. 


With respect to raw milk from goat farms, the Ministry informed us that it has made 
improvements to sample collection and testing procedures. Two additional sample depots 
were established to collect and transport goats’ milk for testing. The Ministry informed us 
that, as a result, the number of goats’ milk producers routinely tested on a monthly basis has 
increased since 2001. For December 2002, the Ministry tested 85% of the producers and 
was hoping to test all producers on a monthly basis during 2003. Unacceptable bacterial 
levels in goats’ milk trigger a visit from the Ministry to the farms to correct the deficiencies 
before further goats’ milk is accepted into the system. The Ministry was developing 
enforcement tools, including penalties, as part of the regulations under the Food Safety and 
Quality Act, 2001, with anticipated implementation in early 2004. 


Horticulture Monitoring 


Recommendation 

To help ensure that plant-based foods are safe from biological and chemical contamination, the 
Ministry should: 

¢ send letters outlining laboratory test results to growers and retailers on a timely basis 


and follow up to ensure that serious safety violations are rectified; and 


¢ develop a risk-based monitoring and inspection process for fruit and vegetable 
commodities to help reduce the risk of outbreaks of illnesses associated with domestic 
and imported horticultural products. 
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Current Status 
When safety violations are noted through testing of horticultural products, the Ministry 


informed us that its current practice is to immediately telephone the grower or retailer 
regarding the health hazard, send a letter to the grower or retailer about the hazard, and 
schedule an advisory visit within a week, so that corrective action can be taken quickly. To 
help with the monitoring and inspection process, the Ministry indicated that it had 
implemented a risk-based system that evaluates hazards through risk profiles of the fruit and 
vegetable commodities and assigns a ranking with regard to the risk. As of May 2003, risk 
assessments had been completed for apple cider, sprouted seeds, lettuce, stone fruit, apples, 
cantaloupe, grapes, cabbage, carrots, and rutabagas. Assessments were underway for other 
fruits and vegetables and greenhouse crops. In addition, the Ministry, along with the 
Ministry of Health and Long-Term Care, participates in the monitoring of outbreaks of 
illnesses associated with domestic and imported horticultural products. In this regard, the 
Ministry uses outbreak information to develop food safety programs to help reduce future 
occurrences, 


Legislative Enforcement 


Recommendation 
To better ensure the safety and quality of Ontario s food supply, the Ministry should: 


° ensure penalties are adequate to deter non-compliance with legislative requirements; 
© review its enforcement rights and responsibilities for consistency; and 


¢ consider periodically following up on the operations of past offenders to ensure 
continued compliance with legislation. 


Current Status 


The Ministry informed us that the average fine imposed during 2002 under the Meat 
Inspection Act (Ontario), the Dead Animal Disposal Act, and the Milk Act was $1,100, 
which was much higher than the $320 we reported in 2001. The Ministry also informed us 
that it had been seeking the maximum penalties provided for under existing regulations. 
Furthermore, it was planning to increase the penalties to $25,000 from the current 
maximum of $5,000 when the regulations under the Food Safety and Quality Act, 2001 are 
introduced in early 2004. When proclaimed, this Act will consolidate six other acts to 
provide: 


¢ a framework for consistent standards and requirements to better protect the public on a 
timely basis from food-borne hazards; and 


¢ increased powers for inspectors to ensure compliance with legislation. 


The Ministry has implemented a pilot monitoring program in co-operation with the federal 
and municipal governments to ensure that past offenders do not attempt to market 
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products processed illegally. The Ministry indicated that this monitoring would be 
expanded when additional funding is allocated to the program. 


Program Co-ordination 


Recommendation 


To better ensure that consistent standards of food safety are in place for all products consumed in 
Ontario, the Ministry should: 


¢ develop appropriate food safety policies for products that are not subject to specific 


legislation and consider incorporating such products in any proposed new legislation; 


° review regulatory requirements to determine the level of risk associated with each 
product and assess the nature of the monitoring required; and 


¢ where monitoring 1s considered necessary, review the inspection process to ensure that 
minimum standards of food safety are maintained regardless of which level of 
government inspects the food products. 


Current Status 


According to the Ministry, it was working with the Ministry of Health and Long-Term Care 
and the Canadian Food Inspection Agency to evaluate risks and develop appropriate food 
safety policies and monitoring programs for all commodities across the production-to- 
consumption spectrum. This includes products not currently subject to specific food safety 
legislation, such as further processed meat outside of licensed slaughter plants, horticultural 
products, sheeps’ milk, and fish. 


The Ministry indicated that it was carrying out a review of meat, dairy, and deadstock to 
develop regulations based on risk under the Food Safety and Quality Act, 2001. The review 
includes a consideration of the risk-mitigation options for other high-risk products such as 
unpasteurized cider. 


The Ministry was developing inspection protocols for verifying compliance with 
requirements and the meeting of minimum food safety standards. The protocols are to 
apply regardless of which ministry, level of government, or third party is delivering the 
inspection service. For example, the Ministry was developing a Hazard Analysis Critical 
Control Point System to identify potential hazards along the food chain; the system is for use 
by non-federally registered, regulated, and unregulated facilities. The protocols should 


ensure that any new regulations will at least meet national standards. 
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MEASURING AND REPORTING ON PROGRAM 
EFFECTIVENESS 


Recommendation 


To ensure that the Food Industry Program can appropriately assess the extent to which it is 
meeting its goals for food safety and increased domestic sales, the Ministry should: 


¢ develop well-defined performance measures that the Ministry has a reasonable degree of 
influence over and can link to expected outcomes; and 


° perform the evaluations necessary to determine whether its activities are effective in 
achieving the stated goals of the Program and to take any necessary corrective action. 


Current Status 


The Ministry indicated that it had developed new performance measures for the Food 
Industry Program. These include the number of incidents of food safety non-compliance 
identified during abattoir audits as a percentage of the total number of standards assessed; 
the percentage of adverse laboratory tests acted on according to prescribed protocols and 
time frames; and the number of incidents of violative levels of food-borne chemicals 
identified in domestic raw foods of plant origin during monitoring activities as a percentage 
of total number of tests. In addition, to determine whether its activities are effective, the 
Ministry intends to complete a thorough evaluation of the food safety program in 2003. 
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4.02—Legal Aid Ontario 
(Follow-up to VFM Section 3.02, 2001 Annual Report) 





BACKGROUND 


The purpose of Legal Aid Ontario is to promote access to justice throughout Ontario for 
eligible low-income individuals. Its primary method of serving clients’ legal needs is through 
the use of legal aid certificates, which allow clients to receive legal representation from their 
choice of private-sector lawyers for a variety of legal problems. During the 2000/01 fiscal 
year, over 107,000 legal aid certificates were issued, and about 5,000 private-sector lawyers 
provided services to legal aid clients. | 


In addition to providing legal aid certificates, Legal Aid Ontario also delivers legal aid 


services through: 


* community legal clinics—in 2003, 79 independent clinics (about 70 in 2000/01) 
specializing in addressing the needs of low-income individuals who need legal help in 


such areas as income maintenance, housing, and access to basic social services; and 


¢ aduty counsel program—a combination of private-sector and staff lawyers providing 
assistance to clients who do not have a lawyer to represent them in the courtroom. 


For the 2002/03 fiscal year, Legal Aid Ontario had operating expenditures of over 
$291.7 million ($247.3 million in 2000/01) and received funding of $282.2 million 
($249.5 million in 2000/01). Of the total funding, contributions from the province 
amounted to $199.4 million in 2002/03 ($171 million in 2000/01), which represented 
about 70% of the total in both years. The other contributors to the funding included the 


federal government, the legal profession, clients, and others. 


We concluded that certain procedures and systems were not in place to ensure that legal aid 
services and programs were provided with due regard for economy and efficiency and in 
accordance with legislative requirements. Some of our more significant concerns were as 
follows: 


¢ To meet the legal needs of low-income individuals cost effectively and to comply with 
the Legal Aid Services Act, 1998, a proper assessment of how those legal needs can best 


be met is required. However, such an assessment had not yet been done. 


¢ The legal aid system had not been effective in controlling the costs of its certificates. 
Annual levels of funding for the four fiscal years from 1996/97 to 1999/2000 were 
similar to the level of funding for the 1991/92 fiscal year. However, two to three times 
more people were provided with legal aid certificates in 1991/92 when compared to 
the number of people served in each of the more recent four years. 
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e Efforts to collect over $100 million of accounts receivable required improvements. 


We made a number of recommendations for improvement and received commitments 
from Legal Aid Ontario that it would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


Based on information received from Legal Aid Ontario, action has been taken to implement 
the majority of our recommendations and action is in progress for the remaining 
recommendations. The current status of all our recommendations is outlined as follows. 


LEGAL AID CERTIFICATION PROGRAM 


Service-Level Requirements and Needs Assessment 


Recommendation 


To comply with legislation and to ensure that the legal needs of low-income individuals are served 
in an appropriate and cost-effective manner, Legal Aid Ontario should perform comprehensive 
and ongoing assessments of service-level requirements and the legal needs of potential clients. 


Current Status 


In 2000/01, Legal Aid Ontario initiated a two-phase needs assessment process. It has now 
completed phase one of the process, which includes a number of components: project 
planning, a clinic needs-assessment project, consultations with clients and service providers, 
needs-based legislative analysis, demographic studies, pilot projects, and a federal needs- 
research program. Phase two of the process, which is still ongoing, includes: second-phase 
needs-assessment strategy, special projects, a pilot/research program, and ongoing needs- 
assessment capacity. 


Legal Aid Ontario further indicated that all the information gathered from the needs- 
assessment process is used to determine the most cost-effective manner of meeting the needs 
of Legal Aid Ontario clients, through, for example, the certificate and duty counsel 


programs. 


Controlling the Cost of Legal Aid Certificates 


Recommendation 


To more effectively manage the cost of legal aid certificates, Legal Aid Ontario should establish 
adequate procedures for monitoring and analyzing lawyers’ billings to establish reasonable billing 
standards and to ensure adherence to those standards. 
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Current Status 


Legal Aid Ontario advised us that it has established a Business Analysis Unit to perform 
ongoing detailed analysis of certificate costs. It has also undertaken a review of the “big-case 
management” program to study factors affecting costs of certificates for cases with budgeted 
costs of over $20,000 each and it has developed methods to manage these costs more 
effectively. 


In addition, as part of the Certificate Management Program established in June 2001, Legal 
Aid Ontario has developed a program to target specific legal aid certificate services to be 
reduced or diverted to duty counsel. Target guidelines have been established for the 
province, for each local office, and for the type of cases handled. Legal Aid Ontario is in the 
process of developing strategies for high-volume offices. As well, it is determining long-term 
strategies to raise awareness and identify the impact of Crown and bail practices on legal aid 
demand and costs. 


Legal Aid Ontario indicated that it plans to develop an improved management information 
system to analyze lawyers’ billings with the implementation of its new information 
technology systems in 2004/05. The Total Service Network information technology project, 
for example, will provide information on utilization rates of tariff maximums for the full 
Legal Aid Ontario caseload. The legal aid “tariff” is a schedule of fees payable to certificate 
lawyers for the services they provide to clients. The tariff sets hourly rates and maximum 
number of hours for which Legal Aid Ontario is willing to pay. In developing this IT 
network, Legal Aid Ontario will assess how much data to collect in a cost-effective manner 
from lawyers about services provided on an ongoing basis, and for which types of cases. 


Review of Hourly Rates 


At the time of our 2001 audit, Legal Aid Ontario had established a Tariff Review Task Force 
and was preparing a business case to submit to the Attorney General later that year. We 
encouraged Legal Aid Ontario to consider the matters discussed in our report regarding the 
participation rate of lawyers and the hourly rates paid by other Canadian jurisdictions as 
part of its determination of an appropriate level of compensation for private-sector lawyers. 


Current Status 


The Tariff Review Task Force completed its business case in November 2001 for the 
consideration of the Attorney General. Subsequently, the province revised the tariff rates for 
certificate lawyers with an increase of 5% effective April 1, 2002. Another increase of 5% 
came into effect on April 1, 2003, bringing current tariffs to between $73.87 and $92.84 
an hour, depending on the lawyer’s experience with legal aid cases. Lawyers working in 
certain designated areas in the North are given additional incentives. 
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ALTERNATIVE SERVICE-DELIVERY MODELS 


Recommendation 


In order to properly assess whether its pilot legal aid projects are cost-effective alternative service 
options, Legal Aid Ontario should ensure that these projects are better designed and managed. 


Current Status 


Legal Aid Ontario indicated it is taking the Provincial Auditor’s findings into account when 
it designs and operates pilot projects. The design and planning of pilot projects is based on: 


* — setting specific objectives related to the service model to be tested; 
° stakeholder consultation on the design; 


* clearly established measures of performance and evaluation at the beginning of the 
project; 


* accurate costing; and 
¢ regular review and evaluation of results. 


According to Legal Aid Ontario, current and recently completed pilot projects are 
conforming to this approach. Projects started prior to our 2001 audit that have recently 
been completed include: Youth Court Counsel, Unbundled Family Services, Family Case 
Management, Expanded Duty Counsel, and Family Law Offices. Projects that were started 
after our 2001 audit and are still in process include: Legal Services for Homeless People, 
Youth Court Duty Counsel Pilot, Community Outreach and Quality Support in Refugee 
Law, and Children’s Aid Society (CAS) Mediation. Legal Aid Ontario expects to complete 
all the pilot projects in the 2003/04 fiscal year. 


ELIGIBILITY FOR LEGAL AID 


Recommendation 
To ensure consistency in granting legal aid services, Legal Aid Ontario should develop 


appropriate guidelines to assist area committees in making decisions about whether or not to 


overrule legal aid area-office assessments of eligibility 


Current Status 
According to Legal Aid Ontario, it completed and produced a revised manual in 


September 2001 to ensure that area committees understand and stay within policy 
guidelines. Area committee decisions and the pattern of appeals granted and refused are 
now reported six times a year to area directors and the senior management committee. Area 
committee practices are also discussed at annual conferences with area directors. There are 
established procedures to report problematic area committee decisions to the vice-president, 
client legal services. 
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CLIENT CONTRIBUTIONS 


Recommendation 


To properly safeguard its assets, Legal Aid Ontario should implement appropriate controls over its 
accounts-receivable system. 


To ensure timely collection of amounts owed, Legal Aid Ontario should follow up on outstanding 
inactive accounts to assess the current financial situation of clients and take appropriate action, 
including collection and/or arranging for new payment agreements. 


Current Status 


Legal Aid Ontario informed us that it launched a major project to overhaul and improve 
control over its accounts-receivable system and collection processes. Legal Aid Ontario 
indicated that it has been developing and implementing operational policies, procedures, 
and programs for managing client contributions in the future. For instance, operational 
recommendations have been developed based on a “best practices” review of Legal Aid 
Ontario area offices. Legal Aid Ontario indicated it would ensure that its future information 
technology system would have the capacity to process collections. Review of performance 
within the collections department has been completed. The role of private agencies in 
support of Legal Aid Ontario collection activities is under review. 


Based on its analysis, Legal Aid Ontario developed a new definition of accounts receivable 
and determined that the amount of money in client contributions outstanding as at 

March 28, 2002, was $103.4 million. Of this amount, approximately $89.9 million 
(66,000 accounts) were secured by liens or other property. The remaining $13.5 million 
was unsecured debt. Further analysis enabled Legal Aid Ontario to identify that 26,000 of 
the 66,000 accounts secured by liens, or $30.9 million of the $89.9 million, were no longer 
receivable. As a result, these accounts were subsequently closed on January 31, 2003. 


Legal Aid Ontario indicated that it has been testing various strategies (such as giving 
discounts and other incentives to clients for paying) to determine how to collect the balance 
in the remaining 40,000 accounts secured by liens. The results of these strategies to collect 
outstanding debt will be used to draft and implement policies to support future collection 
activities. Legal Aid Ontario has started to assess its legal tools for collection of debt by: 
analyzing past payment agreements to determine ability to enforce collection using these 
instruments; reviewing the effectiveness of writs; analyzing impact of writs on policy and 
operations; preparing recommendations; and identifying some “quick fixes” to correct 
certain key problems. 
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EFFECTIVENESS OF LEGAL AID SERVICES 


Recommendation 


To ensure proper accountability for its mandate and the services it delivers, Legal Aid Ontario 
should develop appropriate performance standards, effectiveness indicators, and a quality- 
assurance program. In addition, it should measure and report on its effectiveness in providing 
legal aid services on a timely basis. 


Current Status 
The Legal Aid Services Act, 1998 requires Legal Aid Ontario to provide “consistently high- 


quality legal aid services in a cost-effective manner” to low-income individuals throughout 
Ontario. Legal Aid Ontario indicated that its various departments had developed 
performance standards and effectiveness measures for reporting on their effectiveness in 
providing client services. In addition, Legal Aid had also created a Quality Service Office to 
work with managers, staff, and service providers to develop and implement management 
tools and processes to achieve continuous improvement in the quality of service to clients. 


According to Legal Aid Ontario, 70 community clinics and five Student Legal Aid Services 
Societies (SLASSs) had undergone Legal Aid Ontario’s on-site quality assurance reviews. The 
reviews’ results and recommendations were to be used by the Quality Service Office to 
support the clinics and SLASSs in the continuous development of performance and 
outcome measures in the coming year. Legal Aid Ontario reported that it had completed 
regional consultation with duty counsel regarding quality standards and that consultations 
with lawyers dealing in refugee law were underway. As well, it indicated that it had begun 
discussing with the Law Society of Upper Canada the objectives and approaches common 
to their respective quality assurance programs, as well as identifying areas where Legal Aid 
Ontario and the Law Society of Upper Canada can co-ordinate their efforts and support 
each other’s initiatives. 
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MINISTRIES OF THE ATTORNEY GENERAL 
AND PUBLIC SAFETY AND SECURITY 


4.03-—Integrated Justice Project 
(Follow-up to VFM Section 3.03, 2001 Annual Report) 





BACKGROUND 


After our 2001 Annual Report, the Ministry of Correctional Services was consolidated with 
the Ministry of the Solicitor General in April 2002 to form the Ministry of Public Safety 
and Security. The Integrated Justice Project was instituted in 1996 as a joint initiative of the 
Ministry of the Attorney General and what is now the Ministry of Public Safety and 
Security (Ministries). The objective of the Project was to improve the information flow in the 
justice system by streamlining existing processes and replacing older computer systems and 
paper-based information exchanges with new, compatible systems and technologies. In 
addition, a Common Inquiry System was to be created to allow authorized persons in one 
justice area to access and thus link to files held in other areas on cases, victims, witnesses, 
suspects, the accused, and convicted offenders. The Project was to affect approximately 
22,000 employees in the Ministries at 825 different locations across Ontario, as well as 
municipal police forces, judges, private lawyers, and the general public. 


The Project was implemented using the Common Purpose Procurement (CPP) process, 
under which the government and private-sector partners jointly provide necessary human 
and financial resources and share in resulting risks and rewards. 


The Integrated Justice Project had experienced significant cost increases and delays. While 
the March 1998 cost estimate to complete the Project was $180 million, by March 2001 
the estimate had risen to $359 million. Over the same period, expected benefits were 
reduced from $326 million to $238 million. In addition, not all systems were expected to 
be fully implemented by the contractual deadline of August 2002. We had several concerns 


with respect to these costs increases and delays. 


We concluded in 2001 that the requirement of CPP policy that due diligence be 
performed to support the projections of costs and benefits in a business case was not 
adequately followed in the Integrated Justice Project. We found the following weaknesses in 
the original business case, on which project approval was based, and in subsequent business 
cases used to monitor project progress: 


¢ The original business case had an aggressive schedule that was based on a best-case 
scenario. It did not adequately take into account the magnitude of change introduced 
by the Project, the complexity of justice administration—particularly that of the 
courts—or the ability of vendors to deliver the Project’s computer systems in the 
required time frames. 
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e The estimate of benefits, already reduced to $238 million in the most recent business 
case, was still overstated by approximately $57 million. 


In addition, we noted that no agreement had yet been reached between project 
management and senior management of the courts as to whether or not all of the expected 
courts benefits, totalling $172 million and representing over 70% of the Project’s total 
benefits, would be realizable. 


We also concluded that aspects of the contractual arrangements with the vendor resulted in 
the Project not being administered with due regard for economy. For example, negotiated 
rates for consortium staff were at a premium compared to rates charged by the same vendor 
to other ministries for similar work, increasing total project costs by up to $25 million. In 
addition, the billing rates of consortium staff working on the Project were approximately 
three times higher than those of the Ministries’ staff for similar work. 


As well, we had concerns about the security measures for the systems already in use by police 
and the system to be established for corrections. The confidential information contained in 
these systems—including data on suspects, victims, witnesses, the accused, and convicted 
offenders—was vulnerable to unauthorized access and manipulation. 


We made a number of recommendations for improvement and received commitments 
from the Ministries involved that they would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


Following 20 months of negotiations between the Ministries and the consortium led by 
EDS Canada Incorporated (EDS), the parties were unable to agree on new terms to amend 
and renew their agreement to continue the Project using a Common Purpose Procurement 
approach. As a result, the Project’s Work Term expired on October 8, 2002. Effective 

April 2003, accountability for individual Project components, including fiscal management 
and any future development and/or enhancement, has been assigned to each component’s 
most related ministries. 


The Project’s total investments and benefits as of October 8, 2002 were as follows: 


Total Investments and Benefits of the 
Integrated Justice Project as of October 8, 2002 
($ million) 


ies Consortium | | hes : 


T1102 154.8 265.0 


Se hs oe 


Source of data: Ministry of the Attorney General 
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We were advised that, in addition, both the Ministries and the consortium have incurred 
additional transition costs in connection with the end of the Work Term, which have yet to 


be agreed to by both parties. 


EDS has filed a Notice of Action against the province of Ontario. In addition, the 
government, on behalf of the Project, has initiated a claim seeking damages from a software 
firm for performance deficiencies due to the firm’s delivery failure. 


Based on the information we received from the Ministries, we determined that further work 
was necessary to ensure that those of our recommendations that are applicable to future 
similar projects are implemented so that those projects can benefit from lessons learned. 


BUSINESS CASE 


Recommendation 


Should negotiations result in the Ministries deciding on continuing the Integrated Justice Project, 
the Ministries should take the appropriate measures that will result in completing the Project in 
a timely and cost-effective manner. Controls on project management should be reviewed to 


identify ways to: 


° minimize the risk that the Project will not be completed within the revised timelines 
and to ensure that the revised costs will not be exceeded: and 


° ensure that estimated benefits are, in fact, realizable and ultimately achieved. 


Current Status 


Following the end of the Integrated Justice Project’s Work Term, the Ministries undertook a 
period of reassessment to determine how best to proceed in a post-Project environment. 
Accountability for individual Project components, including fiscal management and any 
future development and/or enhancement for those components, now resides with each 
component's respective Ministry. 


We were advised that initial releases of the Computer-aided Dispatch and Records 
Management systems for the police and the Offender Tracking Information System for 
corrections were implemented by the end of the Project’s Work Term, albeit without all the 
intended functionality, and are now the responsibility of the Ministry of Public Safety and 
Security. The courts applications (Digital Audio Recording, E-File, and Civil and Criminal 
Case Management) and the Crown attorneys’ Case Management System, all of which are 
under the Ministry of the Attorney General, were not completed when the Project’s Work 
Term ended and likely will not be completed as originally envisioned. In addition, one of 
the Project’s primary objectives—that of integrating or sharing data between the various 
justice Ministries’ systems, which would have included implementing a Common Inquiry 
System—was not achieved. 
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We were informed that the Ministries remain committed to modernizing the justice system 
to increase public safety, improve service, and increase access to justice. Any new 
development or system enhancements by the Ministries are to be based on business cases 
that are affordable, staged, and cost-effective and that set realistic time frames. 


PROJECT TIMETABLE 


Recommendation 

To improve controls over timelines and associated costs for this and future Common Purpose 
Procurement projects, the Ministries should work with the Management Board of Cabinet to 
revise Common Purpose Procurement guidelines to require that: 


¢ yeliable information that is validated at the earliest opportunity be used in the 
preparation of the business case; and 


° where significant assumptions must be made, multiple business-case scenarios be 
prepared to help ensure that the financial risks relating to each of the various scenarios 
are clearly presented. 


Current Status 


We were informed that the Management Board Secretariat (MBS) was in the process of 
reviewing the Government’s Common Purpose Procurement (CPP) policies, and that this 
review would consider the Government's experience with large and complex contracts, 
including the Integrated Justice Project, as well as the experiences of other jurisdictions and 
related recommendations contained in the Provincial Auditor’s reports. Once the MBS has 
completed the CPP review, the Government expects to identify ways to enhance its 
procurement policies and guidelines to reflect best practices in public-private partnerships 
for the delivery of services to the public. 


PROJECT BENEFITS 


Recommendation 


To help ensure that benefits identified in the business case for the Integrated Justice Project and 
any future projects are objectively and realistically presented, the Ministries should: 


¢ include in the business case specific details on cost savings and new revenues; and 


¢ ensure that only well-researched and project-specific cost savings and new revenues are 
stated in the business case. 
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CONTRACTUAL ARRANGEMENTS 


Recommendation 


To help ensure that future remuneration to vendors participating in Common Purpose 
Procurement projects is reasonable and fair, the Ministries should work with the Management 
Board of Cabinet to develop appropriate guidelines for the contract negotiation process. These 


guidelines should require that ministries examine ways of reducing the need for large incentives. 


Recommendation 


To ensure that vendor rates charged to the Integrated Justice Project are comparable to the rates 
available to other ministries of the Ontario government and comply with the contractual 
arrangements, the Ministries should renegotiate the vendor s rates. 


In future Common Purpose Procurement projects, to help ensure that the rates being charged can 
be substantiated, adequate provisions for their verification should be included in any contractual 
arrangements. 


Recommendation 


To help ensure that the savings achieved in future Common Purpose Procurement projects are 
fairly distributed, the Ministries should ensure that rates set for their staff are comparable to rates 
used by vendors whenever possible. When rates are not comparable, the Ministries should 
document the justification for the rate differential. 


The Ministries should also take corrective action to ensure that the future staffing costs they 
charge to the Integrated Justice Project are accurately calculated. 


Current Status 


The principles underlying our recommendations continue to be valid. With the Integrated 
Justice Project’s termination, individual ministries were made responsible and accountable 
for any new development or system enhancements at their ministry and will be using 
existing policies for justification and approval of new information technology projects, 
including the further development of former Project components. 


We understand that the above recommendations will be considered during Management 
Board Secretariat’s review of the Common Purpose Procurement process to ensure that 
Common Purpose Procurement guidelines reflect best practices and that enhanced 
procedures will be identified. These procedures could be incorporated into the new 
guidelines for project managers who are considering using the Common Purpose 
Procurement approach in the future. 
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PROJECT ADMINISTRATION 


Recommendation 


To ensure that in future the Integrated Justice Project is administered with adequate internal 
controls and due regard for economy, the Ministries should: 


¢ ensure that charges to the investment pool are adequately verified with supporting 
documents in accordance with contract terms and approvals and take any additional 
measures considered necessary to eliminate duplicate charges and excessive payments; 


¢ establish proper asset controls over its inventory of computer equipment; and 


° ensure that consulting and related services can be demonstrated to have been acquired 
competitively and that payments are made in accordance with contractual terms and 
conditions. 


Current Status 


While negotiations were underway, the Ministries undertook certain interim measures 
within the parameters of the existing arrangements to improve internal controls, such as 
better tracking of invoice payment and improving the validation process for invoice 


payment by the Project’s Project Management Office. 
Following the expiry of the Project’s Work Term, the Ministries initiated a closing audit by 


an external firm of the investments and benefits to assess the appropriateness of charges to 
the Project and the existence of supporting documents for these claims. The audit had not 
yet been completed at the time of our follow-up. 


Since the Project's termination, the Ministries have taken steps to further strengthen 
contract management with respect to information technology related contracts by 
centralizing control of these contracts within the Ministries’ Justice Technology Services. A 
manager has been appointed to oversee the process to ensure compliance with government 
policies and procedures and with ministry-specific delegations of authority, as well as to 
ensure that payments are made in accordance with the contractual terms and conditions. 


SYSTEM SECURITY 


Recommendation 


To help ensure that confidential data in the Integrated Justice Project systems are adequately 
protected against unauthorized access and data tampering, the Ministries should: 


° expedite their plans for implementing cryptography and other controls to secure data 
transmitted over the wide-area network; and 


bd implement more rigorous password controls over user accounts. 
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Current Status 


The Ministries informed us that they have implemented more rigorous password controls 
for the police Records Management System (RMS) and the corrections Offender ‘Tracking 
Information System (OTIS) to protect data confidentiality and security. 


Further security measures, such as cryptography, were scheduled for deployment for the 
police RMS in fall 2003. 


In addition, independent Threat Risk Assessments (TRAs) have been completed for the 
police and corrections systems to determine vulnerabilities and any threats to the systems, 
and to identify mitigation strategies. Security at the data centre, which houses the police and 
corrections systems, was also assessed. The Ministry informed us that changes aimed at 
addressing the concerns raised by the TRAs either had been completed or were scheduled 
for implementation during the 2003/04 fiscal year. 
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MINISTRY OF COMMUNITY, FAMILY 

AND CHILDREN’S SERVICES 

4.04—Support to Community Living 
Programs 

(Follow-up to VFM Section 3.04, 2001 Annual Report) 





BACKGROUND 


The Ministry's Support to Community Living programs provide funding for a wide range 
of community-based support services and prevention strategies for adults and children who 
are disadvantaged or living in poverty. The main objectives of these services are to assist such 
vulnerable individuals to live as independently as possible in their communities and to 
reduce the need for more intrusive and costly institutional care. For the 2002/03 fiscal year, 
ministry expenditures for these programs totalled $183.1 million. For the 2000/01 fiscal 
year, they totalled $155.6 million. 


In 2001, we concluded that the Ministry’s administrative policies and procedures were not 
adequate to ensure that transfer payments were reasonably linked to the quality and level of 
services provided and that the funds were prudently spent for the purposes intended. We 
also concluded that the Ministry did not monitor and assess the services provided by 
transfer-payment recipients to ensure that they were meeting its expectations. In particular, 
we found that: 


¢ Ministry payments for both emergency and domiciliary hostel placements exceeded the 
agreed-upon per diem rates. For instance, over the past three years, the Ministry paid 
one municipality $16.5 million more than it was required to pay for emergency hostel 
stays. 


e Funding for services paid for on a non-per diem basis was not based on a critical 
assessment of funding needs to ensure that the amounts provided were reasonable and 
commensurate with the level and quality of services provided. 


¢ The Ministry had not implemented the governance and accountability framework it 
developed for all of its transfer-payment agencies in 1999. This framework is necessary 


to hold transfer-payment recipients accountable for the prudent use of ministry funds. 


We made recommendations for improvements in a number of areas and received 
commitments from the Ministry that it would take the necessary corrective action. 
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CURRENT STATUS OF RECOMMENDATIONS 


The Ministry’s internal audit services reviewed the status of the Ministry’s actions in response 
to our 2001 audit recommendations. We reviewed its work and determined that we could 
rely on it; we also received additional information from ministry program staff. We found 
from our review of the internal audit work and the other information received that the 
Ministry was in the process of taking action to implement our recommendations. The 
current status of ministry action taken on each of our recommendations is as follows. 


PROGRAM ADMINISTRATION 


Program Funding—Domiciliary Hostels and Emergency 
Hostels 


Recommendation 


To ensure that it is not overpaying for emergency and domiciliary hostel services and that 
payments are made in accordance with the cost-sharing agreement between it and consolidated 
municipal services managers, the Ministry should: 


¢ request information on actual hostel occupancies and on per diem rates and personal- 
needs allowances actually paid by consolidated municipal service managers to support 
monthly claims; 


¢ verify that the monthly amounts claimed and paid do not exceed the maximum 
reimbursable amounts; and 


¢ periodically verify the reliability of the information provided to detect ministry 


overpayments. 


Current Status 


Ministry-wide policies and procedures requiring the verification of occupancy and per diem 
rate information to avoid monthly overpayments have not yet been developed. Nevertheless, 
one of the three regional offices visited in 2001, which accounts for approximately 70% of 
all emergency hostel expenditures, was proactive in addressing this recommendation, since 
the regional office now requests information on actual hostel occupancies, per diems, and 
personal-needs allowances paid in order to verify that the monthly amounts claimed do not 
exceed the maximum reimbursable amount. 


In addition, to reinforce the requirement that claims do not exceed maximums, a ministry- 
wide compliance review and monitoring process is scheduled to begin in the fall of 2003. 
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FUNDING DUPLICATION FOR EMERGENCY HOSTEL RESIDENTS 


Recommendation 


To comply with the Ontario Works Act by ensuring that individuals who reside in emergency 
hostels for extended periods of time are not also collecting Ontario Works benefits, which include 
a shelter allowance, the Ministry should: 


¢ formally assess the extent and impact of such occurrences; and 


° if warranted, require that consolidated municipal service managers identify such 
individuals and develop procedures for taking corrective action where required. 


Current Status 

Although the Ministry has not conducted a formal assessment of the extent to which 
emergency hostel residents were also collecting Ontario Works benefits, a review of 
consolidated municipal service manager practices in the spring of 2002 identified the need 
for clarification in this area. In October 2002, the Ministry sent a bulletin to all 
consolidated municipal service managers, intended to clarify the issue of “funding 
duplication for emergency hostel residents.” 


In addition, we were advised that Service Delivery Model Technology system changes to 
support policy requirements are scheduled for implementation in late 2003. 


Program Funding—Homelessness Initiatives 


Recommendation 
To ensure that funding for homelessness initiatives is spent prudently and in the most effective 
manner for meeting the needs of the homeless, the Ministry should ensure that: 


° funding is approved and provided on a timely basis and is consistent with the 
recipients ability to provide the services agreed to; and 


° ministry funding is used only for the purposes intended. 


Current Status 

The Ministry stated that it continues to be committed to work within the government’s 
funding approval process and to flow resources in a timely manner that reflects the ability of 
consolidated municipal service managers to provide services. 
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Program Funding—Interpreter/Intervenor and Children’s 
Services | 


Recommendation 


To help ensure that ministry funding is reasonable and commensurate with the underlying 
services to be provided and that value for money is being received for services rendered, before 


funding is provided the Ministry should: 
° require that agency budget submissions contain sufficiently detailed information on the 


services to be provided and the related costs to be incurred to enable informed funding 
decisions; and 


¢ critically review and assess the reliability of that information. 


Current Status 


In most cases, agency budget submissions still lack sufficiently detailed information on the 
services to be provided and the related costs to be incurred to enable the Ministry to make 
informed funding decisions. However, the Ministry has initiated training sessions with 
program supervisors to help clarify expectations and their responsibilities in this area. 


We also understand that the Ministry is in the process of examining variances in costs for 
interpreter and intervenor services across the province. This may help to address the issue of 
funding inequities noted in our 2001 Annual Report. 


ANNUAL PROGRAM EXPENDITURE RECONCILIATION 


Recommendation 

To improve the effectiveness of the Ministry's Annual Program Expenditure Reconciliation 
(APER) process for identifying inappropriate or ineligible expenditures, returning excess program 
funding to the Ministry, and supporting future funding decisions, the Ministry should ensure 
that: 


¢ APERs contain sufficiently detailed and relevant information; and 


¢ all surplus funds identified are returned to the Ministry on a timely basis as required by 
Management Board of Cabinet directive. 


Current Status 

The Ministry's 2001/02 Annual Program Expenditure Reconciliation (APER) package 
provided to agencies included updated definitions of admissible and inadmissible 
expenditures. 


However, the Ministry still does not ensure that the APERs and audited financial statements 
submitted by agencies include sufficiently detailed and relevant information to improve the 
effectiveness of the APER process. Therefore, the Ministry is still not in a position to 
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effectively identify both inappropriate and ineligible expenditures. We were advised that 
staff training to improve the APER review and approval process began in the 2003/04 fiscal 


year. 


With respect to excess funding, primarily surpluses self-identified by agencies were being 
returned to the Ministry within 24 months of each agency’s year-end, in accordance with 


ministry policy. 


ACCOUNTABILITY FRAMEWORK FOR AND 
GOVERNANCE OF TRANSFER-PAYMENT AGENCIES 


Recommendation 


To help ensure that services provided are of an acceptable and reasonably consistent standard 
and represent value for money spent, the Ministry should implement and communicate to its staff 
an accountability framework that satisfies the mandatory requirements of the Management 
Board Directive on Transfer Payment Accountability 


To enhance and justify the reliance the Ministry can place in the cost-effective governance of 
transfer-payment recipients, the Ministry should ensure that the conditions for such reliance have 
been communicated and are in place. 


Current Status 


The Ministry’s Governance and Accountability: Framework for Transfer Payments to 
Community Agencies was approved in April 2003. This document will be accessible to 
community agencies through the Ministry’s Web site, as well as linked to the Ministry’s 
transfer-payment budget package for the 2004/05 fiscal year. This framework outlines the 
basic requirements and required activities for ministry staff and agencies. We were also 
advised that a resource manual for boards of directors was being developed and is expected 
to be distributed to transfer-payment agencies in early 2004. 


Beginning in the 2001/02 fiscal year, the Ministry required agencies to affirm that their 
boards of directors have the expertise and experience necessary to discharge their 
responsibilities and that they also have an appropriate governance and reporting structure 
in place. 
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MINISTRY OF COMMUNITY, FAMILY 
AND CHILDREN’S SERVICES 


4.05-Violence Against Women Program 
(Follow-up to VFM Section 3.05, 2001 Annual Report) 





BACKGROUND 


The Ministry’s Violence Against Women (VAW) program funds transfer-payment agencies 
that provide safe shelter and other support services to women who have experienced 
violence or abuse as well as to their children. 


For the 2002/03 fiscal year, ministry spending on this program totalled approximately 
$90.7 million, with ministry spending on this program being approximately $82 million for 
the 2000/01 fiscal year. For 2000/01, the Ministry provided approximately $64 million to 
nearly 100 community-based non-profit agencies that operated shelters providing 
temporary accommodations and security to approximately 15,000 women and 13,000 
children during the year. The Ministry also provided approximately $18 million during that 
year to more than 100 other community-based non-profit agencies that did not operate a 
shelter but provided other support services such as counselling, violence prevention, and 
public education programs. 


At the time of our 2001 audit, we concluded that the Ministry's monitoring and assessment 
practices did not ensure that the services provided by the transfer-payment agencies we 
reviewed were of an acceptable and consistent quality standard or that they represented 
value for money spent. We also concluded that the Ministry’s policies and procedures were 
not adequate to ensure that transfer payments to agencies providing services were in all cases 
reasonable and sufficiently controlled. In particular, we found that: 


¢ In some cases, women and children were turned away from shelters, and waiting times 
for other services were lengthy. 


¢ The amount of funding provided to transfer-payment agencies was not based on an 
assessment of what costs would be reasonable for the services to be provided. As a result, 
the cost of similar services varied significantly among agencies. 


¢ The Ministry’s annual process for reconciling an agency’s actual expenditures against 
funds provided was in many cases deficient in identifying inappropriate or ineligible 
expenditures as well as funding surpluses that should have been recovered. 


We made recommendations for improvements in each of these areas and received 
commitments from the Ministry that it would take the necessary corrective action. 
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CURRENT STATUS OF RECOMMENDATIONS 


The Ministry's internal audit services reviewed the status of the Ministry’s actions in response 
to our 2001 audit recommendations. We reviewed their work and determined that we 
could rely on it. We found that the Ministry has made limited progress on implementing 
our recommendations. The current status of ministry action taken on each of our 
recommendations is as follows. 


MONITORING OF SERVICES PROVIDED 


Service Quality Standards 


Recommendation 


10 ensure that services provided by all transfer-payment agencies under the Violence Against 
Women program are of an acceptable and reasonably consistent quality standard and that they 
represent value for money spent, the Ministry should: 


° establish and communicate its expectations for acceptable quality standards of service 
in each of the following areas: core services to be provided: minimum staffing levels and 
staff qualifications; admission criteria; acceptable waiting times; the physical security of 
shelters; and service co-ordination with other providers; and 


° periodically assess whether the services provided by agencies are meeting ministry 
expectations and take appropriate corrective action if necessary, 


Current Status 


In February 2002, the Ministry sent out a letter to all regional offices that included a 
statement of what a shelter’s core services should be, admission criteria, and an overview of 
the serious occurrence reporting procedures. In addition and as a result of its hearings into 
our 2001 report, in December 2002 the Standing Committee on Public Accounts 
requested that the Ministry report on its progress with respect to defining core services and 
developing a policy to clarify the services not offered by shelters. In February 2003, the 
Ministry responded to this request by providing a listing of the major components of a 
shelter’s service and other ancillary services that may be offered by a shelter. 


The Ministry advised us that it is currently reviewing shelter staffing information and will 
assess the feasibility of developing a program guideline regarding staffing levels and core 
competencies. 


The Ministry also developed and distributed the Children’s Aid Society/VAW Shelter 
Collaboration Agreement Template and Guidelines to promote collaboration between the 
two sectors. The Ministry anticipates that agreements between them'will be implemented 
across the province by spring 2004. 
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The issue of acceptable standards with respect to acceptable waiting times and physical 
security of shelters has not yet been specifically addressed. However, the Ministry advised us 
that it has allocated $5 million to enhance the safety, accessibility, and security of shelters. 


Only at such time that acceptable quality standards of service have been established and 


communicated can compliance with them be assessed. 


Quarterly Reporting 


Recommendation 


To ensure that the quarterly reporting process for the Violence Against Women program enables 
effective monitoring of expenditures and service delivery during the year, the Ministry should: 


° provide direction to all agencies to ensure that they report financial and service 
information on a consistent basis; 


° ensure that agencies explain and describe appropriate plans of action for all significant 
variances, as required; and 


° promptly evaluate and, where warranted, approve any corrective action proposed and 
ensure that it is undertaken. 


Current Status 


In order to assist regional offices to more effectively monitor agency expenditures and service 
delivery, the Ministry revised and expanded the type of information to be reported by 
agencies. For instance, the Ministry introduced the Transfer Payment Business Cycle 
Checklist in September 2001. In February 2002, the Ministry sent a letter to regional 
offices reinforcing the use of this checklist. 


However, the Ministry’s internal audit services found that this checklist was either not being 
used by ministry staff, or when it was used, it was not fully completed. By not using the 
checklist or by not fully completing it and documenting the necessary analysis, the Ministry 
did not have the assurance that its business practices were being followed by the agencies 


being funded. 


Serious Occurrences 


Recommendation 
To enhance the effectiveness of the serious-occurrence reporting-and-review process for agencies 
funded under the Violence Against Women program, the Ministry should ensure that: 
° agencies have a clear understanding of how the criteria for serious occurrences are to be 
applied; 


¢ all verbal notifications of serious occurrences are logged so that it can reliably determine 
whether written follow-up reports are received as required; and 
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° appropriate corrective action 1s taken as a result of all serious occurrences that take 


place. 


Current Status 

As of April 1, 2003, new forms for the reporting of serious occurrences were implemented. 
The forms included an area to document the name of the ministry employee who received 
initial notification of the serious occurrence and how the employee acknowledged the 
receipt of this information. The Ministry’s internal audit services examined the serious- 
occurrence report process in three regional offices. It found that the forms were being used 
as intended and that these offices logged all serious occurrences to ensure that the reporting 
process was adhered to and timely follow-up occurred, if necessary. 


TRANSFER-PAYMENT AGENCY ACCOUNTABILITY AND 
GOVERNANCE 


Recommendation 


The Ministry should improve the governance and accountability of Violence Against Women 
agencies by implementing the requirements of its Governance and Accountability Framework. 


Current Status 


Beginning in the 2001/02 fiscal year, the Ministry required agencies to affirm that their 
boards of directors had the expertise and experience necessary to discharge their 
responsibilities and that they also had an appropriate governance and reporting structure in 
place. 


We were advised that, as of May 2003, the Ministry had finalized a governance and 
accountability framework for transfer-payment agencies, but it had yet to send the 
framework out to the agencies. We were also advised that a resource manual for boards of 
directors was being developed and was expected to be completed in fall 2003. 


PROGRAM FUNDING 


Funding for Shelter Services and Funding for 
Counselling and Other VAW Services 


Recommendation 
Lo help ensure that funding for services under the Violence Against Women program is 
commensurate with the specific needs of each agency, the Ministry should: 


° ensure that agency funding requests provide information that is sufficiently detailed 
and relevant to allow the Ministry to make informed funding decisions; and 
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¢ critically assess all requests for funding and ensure that amounts approved are 
commensurate with the demand for services and the actual services provided. 


Current Status 


The Ministry’s internal audit services examined a sample of ministry files and found that the 
funding requests still did not contain sufficiently detailed and relevant information to allow 
the Ministry to make informed funding decisions. Therefore, the Ministry has yet to address 


this recommendation effectively. 


Annual Program Expenditure Reconciliations 


Recommendation 


To better ensure that the Annual Program Expenditure Reconciliation (APER) process for the 
Violence Against Women program identifies both inappropriate and ineligible expenditures, as 
well as surpluses, and better supports future funding decisions, the Ministry should: 


¢ clearly define eligible program expenditures and communicate that information to 
Aagenctes; 


° ensure that agencies provide financial information in their APERs that is sufficiently 
detailed and linkable to audited financial statements; and 


¢ thoroughly review the information submitted in APERs and perform any necessary 
follow-up. 


Current Status 


The Ministry acknowledges the need for strong financial accountability processes and has 
taken steps to improve the Annual Program Expenditure Reconciliation (APER) form and 
process. For example, the Ministry's 2001/02 APER package provided to agencies included 
updated definitions of admissible and inadmissible expenditures. 


However, the Ministry still does not ensure that the APERs submitted by agencies include 
sufficiently detailed and relevant information to improve the effectiveness of the APER 
process. Therefore, the Ministry is still not in a position to identify both inappropriate and 
ineligible expenditures. With respect to excess funding, surpluses self-identified by agencies 
were being returned to the Ministry within 24 months of each agency’s year-end, in 
accordance with Ministry policy. 
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MINISTRY OF EDUCATION 


4.06—Special Education Grants to School 
Boards 
(Follow-up to VFM Section 3.06, 2001 Annual Report) 





BACKGROUND 


About 277,000 of the more than 2 million students attending Ontario’s publicly funded 
schools receive special education programs and services. The strengths and needs of students 
with special education needs vary widely, from gifted at one extreme to those requiring very 
intensive support at the other. 


For the school year ended August 31, 2002, the Ministry of Education provided 

$1.37 billion in special education funding to school boards (up from $1.36 billion in 
2000/01). For the 2002/03 school year, it provided about $1.62 billion. These grants are 
intended to cover only the incremental costs of providing special needs assistance. The 
common basic costs of educating all students, including students with special education 
needs, are covered by other grants. Although the Ministry is ultimately accountable for the 
quality of Ontario's education system, it relies heavily on school boards as delivery agents to 
design and deliver quality programs and services to students. 


In our 2001 audit, we examined the Ministry’s administration and oversight of special 
education grants and services and visited selected school boards to review and discuss their 
special education expenditures and service-delivery practices. At that time, we concluded 
that neither the school boards we visited nor the Ministry, which was in the process of 
implementing a multi-year plan to strengthen accountability for special education grants 
and services, had the information and processes in place to determine whether special 
education services were being delivered effectively, efficiently, and in compliance with 
requirements. Our observations included the following: 


¢ Individual Education Plans (IEPs) that we had reviewed did not meet either regulatory 
requirements or ministry expectations. IEPs are essential if each student is to receive the 
special education he or she requires. 


* Neither the Ministry nor the boards had established quality assurance processes to 
ensure that suitable programs and services were being delivered to students with special 
education needs. 


* School boards did not collect and report sufficient, appropriate.information on their 
special education expenditures and service delivery to support decision-making by 
school-board management and to enable effective oversight by the Ministry, trustees, 
and parents. 
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¢ Many educators expressed concerns about insufficient numbers of teacher assistants and 
special education resource teachers to help them meet the needs of their students. 


We had also reviewed the status of the 15 recommendations made in 1994 by the Standing 
Committee on Public Accounts as a result of our previous audit of special education 
programs and services. The Ministry had implemented four, was making progress on 
implementing six, and had not addressed five. 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


For several years the Ministry has been introducing major changes to Ontario's education 
system that are intended to improve its equity, quality, efficiency, and accountability. Many 
of these changes, such as the new curriculum, provincial report card, and provincial testing, 
are intended to benefit all students. The most significant changes aimed specifically at 
programs and services provided to students with special education needs began to be 
implemented in 1998. For example, standards for the preparation of Individual Education 
Plans and for the public reporting of school-board Special Education Plans were provided 
to school boards in 2000. As well, initiatives to set standards for the administration and 
delivery of special education programs and services were announced in January 2000 and 
were planned for completion by the end of 2002. 


As of June 2003, although the Ministry has made progress, the completion of a number of 
major initiatives currently underway is needed before the Ministry and school boards will 
have assurance that special education funds are being spent in accordance with ministry 
expectations and students with special education needs are getting the programs and 
services they need. 


We also reviewed the status of recommendations made by the Standing Committee on 
Public Accounts in 1994 and noted that significant progress in addressing the Committee's 
recommendations had been made since our 2001 audit. 


The status of each of our 2001 recommendations is outlined below. 


INDIVIDUAL EDUCATION PLANS 


Compliance with Guidelines and Standards 


Recommendation 


To help ensure that the services students need to make satisfactory progress are timely and 
appropriate and that school boards comply with legislation, regulations, and policies, the 
Ministry should: 


Special Education Grants to School Boards 301 





Follow-up Section 4.06 


Follow-up Section 4.06 





¢ use the results of their planned compliance-verification work to provide boards with 
examples of successful practices for timely and effective preparation of Individual 
Education Plans; and 


° require school boards to establish compliance-monitoring procedures and report the 
results of their work. 


Current Status 


The Ministry has substantially implemented the first part of this recommendation. Since our 
audit the Ministry has completed compliance reviews of a sample of Individual Education 
Plans (IEPs) in each of the province’s 72 school boards. Based on the results of the reviews 
conducted at the first 50 boards, the Ministry issued, in September 2002, further guidance 
to clarify certain requirements that were not being met by the boards and developed and 
distributed a template to assist boards to meet ministry expectations more effectively and 
consistently, 


The second part of this recommendation remained to be implemented and is of continuing 
concern given the results of the Ministry’s compliance reviews. The results of the 72 
completed reviews were consistent with our findings in 2001. According to the letter that 
the Ministry sent to all school boards in June 2003, which summarized the 2003 review 
results for the remaining 22 boards, “the greatest concern continues to be that many IEPs 
do not effectively describe educational programming which is clearly connected to the 
Ontario curriculum.” The Ministry also found, for example, that many IEPs lacked 
specificity in a number of areas, such as program goals, strategies, assessment methods, 
accommodations, and the specific knowledge and skills to be assessed each term; that most 
IEPs reviewed did not clearly demonstrate that parent/student consultation had actually 
occurred; and that inappropriate use of terms in most IEPs reviewed resulted in an inability 
to communicate the connections between a student’s IEP and subsequent report card. 


Although the Ministry already requires school boards to develop a plan and to publicly 
report on how they will comply with the Ministry’s requirements for IEPs, the Ministry was 
planning to issue more specific policy directives to school boards regarding quality assurance 
mechanisms for IEPs in fall 2003. 


Supporting Service Decisions 


Recommendation 


To help ensure that educators have the information they need to determine how to best meet the 
needs of students and to help parents assess the adequacy of services and supports, the Ministry 
should require boards to summarize the rationale for key service decisions in Individual 
Education Plans and provide explanations for cases in which planned progress has not been 


achieved. 
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Current Status 


The Ministry was still in the process of implementing this recommendation. The Ministry 
requires that school boards include information that would support service decisions, such 
as assessment results, in Individual Education Plans (IEPs); however, according to the 
Ministry's compliance review results, these requirements were not being met by school 
boards. This recommendation will not be fully implemented until the Ministry determines 
that all boards have established effective quality assurance mechanisms for their [EPs in 
accordance with our first recommendation. 


Reporting on Student Progress 


Recommendation 


To help ensure that all parents are provided with clear reports on their children’s progress, the 
Ministry should provide boards with examples of good practice in reporting the progress of 
students working towards modified learning expectations. 


Current Status 


The Ministry was in the process of implementing this recommendation. The additional 
guidance provided to school boards in September 2002 clarified requirements for clearly 
articulating goals and modified learning expectations for students. However, practical 
examples of good practice had not been provided to boards, and Individual Education 
Plans (IEPs) reviewed in 2003 still did not meet ministry expectations in this regard. The 
Ministry communicated to Directors of Education in June 2003 that it intends to develop 
and distribute further resource supports through the revision of Special Education: A Guide 
for Educators (Part E) and through samples of completed IEPs. These supports are to be 
made available for use during the 2003/04 school year. While every IEP is individualized, 
samples of well-completed IEPs will support better completion of individual IEPs that will 
serve as an instrument to help measure a student's progress. 


PROGRAM QUALITY STANDARDS AND ASSURANCE 


Delivering the Individual Education Plan and Early 
Identification of Student Needs 


Recommendation 


To help ensure that students receive timely and effective programs and services in accordance with 
ministry program standards and expectations, the Ministry should: 


° require and assist boards to implement quality-assurance information systems and 
procedures for their special education and early intervention programs and services; 


and 
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° periodically assess whether the systems and procedures established by boards are working 
as intended. 


Current Status 


The Ministry did not yet have any information on the extent to which school boards had 
taken action to collect better information on their special education programs and services 
and how well they were working. Any improvements may be evident in the Special 
Education Plans that school boards are required to prepare and make publicly available 
every other year, but updated plans were not due from the boards until July 31, 2003. 


Since our audit, the Ministry has introduced an early reading and early math strategy to 
further support school boards to identify and assist, as soon as possible, students who are 
having difficulties in reading and math. The Ministry has also contracted with the Learning 
Disabilities Association of Ontario to develop and pilot a screening tool and targeted 
interventions for children in Kindergarten and Grade 1, to be used by teachers to identify 
and address reduced readiness to learn. Twenty school boards were involved in a pilot that 
concluded in June 2003. The project focused on reading and social readiness. A math 
readiness tool is to be developed in this fiscal year. The Ministry intends to review the results 
of the pilot and make any necessary amendments before releasing the screening tool and 
targeted interventions more widely. This project complements the Ministry’s Early Math 
and Early Reading initiatives. 


One of the major reforms the Ministry announced in January 2000 was the intention to 
develop program standards in order to achieve consistent quality in the special education 
programs and services delivered by school boards. The Ministry had planned to have 
standards for all 12 exceptionalities developed and released by 2003. In May 2003, the 
Ministry released proposed program standards that would apply to the planning and 
delivery of special education programs for all exceptional pupils. 


In addition, the Ministry has initiated a pilot project involving 10 school boards (English 
and French, public and Catholic, representing all regions of the province) to field test the 
standards-based approach to the planning and delivery of special education, using the 
autism-specific standards as a vehicle for this test. The pilot project is intended to identify the 
key activities and changes that are needed to ensure quality improvement in program and 
service delivery and to support effective implementation of all program standards. Program 
standards will not be finalized and released until the pilot project has been completed and 
evaluated. 


Tracking Student Achievement and Outcomes 


Recommendation 


To help ensure that the Ministry and school boards can evaluate the effectiveness of special 
education programs and services, the Ministry should: 
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* require school boards to summarize the progress made by students with special needs 
relative to planned expectations and to report board-wide results on the extent to 
which expectations were met; and 


* establish procedures to collect information on the post-school outcomes of students with 
special needs and report this information at both the school board and the provincial 
level. 


Current Status 


Given the weaknesses in Individual Education Plans that the Ministry’s compliance review 
process identified, little progress had been made on implementing this recommendation. 
The school boards that the Ministry reviewed were not capable of summarizing the progress 
being made by their students with special education needs either individually or collectively. 


The Education Quality and Accountability Office (EQAO) has provided school boards 
with performance results on the 2002 Ontario Secondary School Literacy Test (both the 
Pebruary and October applications) for their students with special education needs. These 
data include exceptionality-specific results and data related to exemptions, deferrals, and 
type/frequency of accommodations. A provincial roll-up and presentation of these data 
were provided to the Minister's Advisory Council on Special Education following the release 
of the results to school boards and were used to help the Ministry’s At Risk Working Group 
to form its recommendations. These recommendations address issues of achievement for 
students at risk of failure in school, including students with special education needs, and 
were being implemented at the time of our follow-up. 


The EQAO has also expanded its guidance to educators who administer provincial 
assessments. This guidance may make these assessments more useful for tracking the progress 
of students with special education needs, provided that the results can be segregated into the 
proportion of students with special education needs in each school board and school who: 


¢ are working towards the Ontario curriculum learning expectations for their grade and 
are identified as special needs only because they require certain accommodations; 


¢ are working towards modified learning expectations in one or more subjects/courses; 
and 


¢ are exempted from the assessments, including the proportion who are working towards 
alternative expectations to the Ontario curriculum. 


According to the Ministry, a number of the Special Education Advisory Committees of 
district school boards have used the data from EQAO tests relating to students with special 


education needs as the basis for discussions about the students’ performance. 


As further support to our recommendation, in September 2002, the Minister's Advisory 
Council on Special Education recommended that the Ministry invest in applied special 
education research in order to develop the capacity of school boards for cost-effective 
programming and planning. The Council concluded that “efficient and effective use of 
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resources cannot be achieved unless school boards have access to research that demonstrates 
how programming can be delivered for maximum impact.” 


By September 2003, the Ministry plans to have assigned a unique Ontario Education 
Number to each student that, together with other information systems projects underway, 
will help facilitate the collection of data on the progress and outcomes of all students, 
including those with special education needs, as well as the analysis of board-by-board 
achievement and change. 


Provision of Professional Services 


Recommendation 


To help ensure that students with special needs have available to them, in accordance with the 
Education Act and regulations, appropriate professional services regardless of the area of the 
province in which they reside, the Ministry should: 


° require that boards collect and report complete service-backlog information; and 


° resolve the co-ordination of services issue with the Ministry of Health and Long-Term 
Care. 


Current Status 


The Ministry was in the process of implementing this recommendation. School boards are 
required to include service-backlog information in their Special Education Plans but, as 
stated earlier, these plans are not due until July 31, 2003. Therefore, the Ministry will not 
know the extent of service backlogs until those plans are received and analyzed. 


The Ministry did provide school boards with $10 million of additional funding, to be spent 
by March 31, 2003, for reducing or eliminating any backlog in professional assessments 
their students required. The funding was to be used first to assess high-need students who 
may qualify for additional funding for support and other resources. 


In March 2002, the Ministry established a Coordinated Services Advisory Committee that, 
in turn, has established several working groups to address specific services for children with 
special education needs, such as planning for entry to school, speech-language services, 
occupational therapy, physiotherapy, psychosocial services, and adaptive technology. 
Representation on the Committee includes individuals from other affected ministries, key 
health service areas, school boards, and parent groups. A recent focus has been on 
establishing performance indicators for a co-ordinated services approach that will permit the 
Ministry to monitor the impact of process improvements while giving school boards the 
flexibility to adapt approaches to local circumstances. 
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MANAGING PLACEMENT AND CLASS COMPOSITION 


Recommendation 


To help support resource allocation decisions and assist trustees and the Ministry to monitor the 
classroom support available to teachers and students, the Ministry should: 


° require that boards collect and report information on class composition and the 
support resources available for each school; and 


° analyze the information to determine how pervasive support or composition problems 
are and take corrective action where necessary. 


Current Status 


The Ministry only recently began to implement this recommendation. In January 2003, the 
Ministry launched the Special Education Data Project to address concerns about the 
reliability and usefulness of information being reported by school boards. The project is 
examining: the relevance of data currently collected to ministry decision-making and 
accountability for special education, the gaps and overlaps between data sources, and the 
comparability of data between the various school boards. One of the project's objectives is to 
improve the organization, quality, and comparability of the Ministry’s information on special 
education students, staff, programs, and services. The Ministry expects to have 
recommendations by fall 2003 and to implement new or revised data requirements in time 
to start collecting information for the 2004/05 school year. 


PLANNING AND OVERSIGHT OF PROGRAMS AND 
SERVICES 


Recommendation 


To help ensure that school-board special education programs and services are effective and to 
enable the Ministry to obtain assurance of that effectiveness, the Ministry should improve tts 
standards for special education plans to include the requirements that: 


¢ trustees, with the advice of Special Education Advisory Committees, establish service- 
delivery objectives for management; 


° management report annually on the extent to which service-delivery objectives have 
been achieved and on any necessary corrective action; and 


¢ boards have systems and procedures to ensure the accuracy and completeness of the 
information presented in the plans and management reports. 


Current Status 


The Ministry decided to delay action on this recommendation until it had reviewed the 
school-board Special Education Plans due in 2003. The Ministry's Standards for School 
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Boards’ Special Education Plans were issued in September 2000 and became effective for 
the submission of boards’ plans in 2001. The next set of board plans were not due to the 
Ministry until July 31, 2003. Once an analysis of the school boards’ Special Education Plans 
is completed, the Ministry was planning to seek advice from the Minister’s Advisory Council 
on Special Education regarding changes to the standards for future submissions. 


TEACHER PREPAREDNESS 


Recommendation 


To help ensure that teachers are well prepared to meet the needs of students with special needs, the 
Ministry should: 


° work with applicable stakeholders to review the pre-service practical experience and 
special education course content requirements for certifying teachers and ensure that 
they reflect best practices in preparing teachers for their responsibilities; and 


° investigate the feasibility of providing a cost-effective, Web-based reference facility for 
educators and of acquiring or developing age-appropriate instructional materials. 


Current Status 


Significant progress has been made on this recommendation. The Ontario College of 
‘Teachers has completed regulatory revisions to the special education component of pre- 
service programs that are intended to ensure that all certified teachers have training in 
instructional strategies designed to meet the individual needs of students, including students 
with behaviour, communication, intellectual, physical, or multiple exceptionalities. 
Although the revised regulation has not yet been passed, some faculties of education have 
already introduced new pre-service experience requirements for special education into their 
teacher training programs. The new qualifying test for teachers also includes content on 
teaching students with special education needs. 


In addition, on December 12, 2002, the Ontario College of Teachers was granted 
regulatory authority to accredit teacher education programs. Accreditation is intended to 
ensure more consistent quality and adherence to professional standards by institutions 
offering such programs. 


The College has also completed major revisions to the additional qualification courses 
teachers must take as part of their new five-year mandatory professional learning program 
beginning in 2003, and special education courses are being significantly expanded as part 
of this initiative. 

Also in 2003, the Ministry released an updated CD-ROM version of the Curriculum Unit 
Planner for teachers that includes a Special Education Companion. ‘The Companion 
includes sample teaching strategies, suggestions, and accommodations for most 
exceptionalities, together with Internet links to Web-based information and resources. 
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FUNDING FOR SPECIAL EDUCATION 


Recommendation 


To strengthen the ability of trustees, Special Education Advisory Committees, parents, and the 
Ministry to hold school-board administrators accountable for spending special education funds in 
a cost-effective manner and to strengthen the Ministry s ability to ensure the adequacy and equity 
of the special education funding formula, the Ministry should: 


¢ require that boards report the full cost of special education by major activities and 
functions and establish standards to ensure that the reported information 1s comparable 
across boards; 


¢ obtain assurance regarding the reliability of the reports; and 


¢ analyze the reported information, determine the reasons for any significant 
inconsistencies in expenditures, and take action where appropriate. 


Current Status 


The Ministry was in the process of implementing this recommendation. The Ministry 
concluded its process for validating the number of high-needs students requiring intensive 
supports in each school board, and final funding adjustments had been made. 


As part of the Special Education Data Project already noted, the Ministry worked with 
school boards to resolve issues relating to the reliability and comparability of expenditure 
data. In April 2003, it issued new special education expenditure reporting instructions to 
school boards that clarify special education expenditure information requirements and 
definitions. The new requirements apply beginning with the 2003/04 school year. 
Consequently, the Ministry will not have more reliable and comparable information on 
school boards’ special education expenditures until fall 2004. Efforts to obtain program and 
activity level expenditure data will take even longer as many school boards do not have the 
necessary systems in place to provide such data. 
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MINISTRY OF FINANCE 


4.07—Community Reinvestment Fund 
(Follow-up to VFM Section 3.07, 2001 Annual Report) 





BACKGROUND 


Effective January 1, 1998, in an initiative that became known as Local Services Realignment 
(LSR), responsibilities relating to 16 government programs and some $3 billion in program 
costs were realigned between the province and Ontario’s municipalities. The programs 
included Municipal Transit, Public Housing, Social Assistance, Public Health, Policing, and 
Land Ambulance services. To help municipalities pay for the programs transferred to them, 
the province took over funding of approximately $2.5 billion in education costs that 
previously had been funded by municipalities from local property taxes. As the 
municipalities were allowed to retain all local property taxes collected, this created what is 
referred to as “tax room” for municipalities. 


The Community Reinvestment Fund (CRE) was established in 1998 with the objective of 
ensuring that the LSR initiative was and remains revenue neutral to municipalities by 
annually providing payments making up the difference between net LSR costs transferred 
and municipal tax room. Since 1998, CRF payments to eligible municipalities have totalled 
approximately $3.1 billion, with $623 million being paid in the 2002/03 fiscal year. In the 
2000/01 fiscal year, $561 million, which included the CRF Bonus and Supplementary 


Assistance, was paid. 


In our 2001 Annual Report, we concluded that the Ministry did not have adequate 
procedures to measure and report on whether the CRF was meeting its revenue neutrality 
objective. In addition, we found that the CRF did not ensure the ongoing revenue 
neutrality of the LSR initiative, either as a whole or for individual municipalities, and that 
this problem had been growing over time. The divergence from revenue neutrality was in 
both directions, with some municipalities clearly gaining from the LSR and others losing. 
We concluded that the CRF as structured at the time of our audit was working against the 
objective of ensuring revenue neutality. Specifically: 


* Eligible LSR costs for fully transferred programs were frozen at the amounts existing at 
the time of program transfer. Accordingly, actual costs incurred by municipalities in 
subsequently delivering these programs were not being taken into account in 
determining CRF entitlements. 


¢ The CREF allocation formula takes into account only those LSR costs that remained after 
the deduction of approximately $500 million annually to reflect a provincially imposed 
savings target. That target is a percentage of total municipal spending that varies 
according to the size of a municipality, and the Ministry had insufficient empirical or 
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analytical support for this approach. Furthermore, since $1.3 billion in LSR programs 
were still administered by the province, the savings target presented municipalities with 
the challenge of finding savings in programs that they did not control. 


¢ Because of the intricacies of the CRF funding formula, savings targets had had no effect 
on some 72 municipalities that experienced annual windfall gains from the LSR 
initiative. Other municipalities experienced a significant, negative fiscal impact. 


¢ The Ministry did not update the residential education tax-room component of the CRF 
payment formula to reflect recent changes in assessment data, including changes arising 
from the latest province-wide current value assessment. Updating the tax-room 
component of the CRF funding formula would have increased the CRF entitlement of 


some municipalities and decreased the entitlement of others. 


With respect to program administration, while we concluded that overall system controls 
and procedures were adequate to ensure that CRF payments were properly authorized and 
processed, we recommended that the Ministry improve its monitoring of municipal use of 
CRF funds, implement procedures to recover or minimize CRF overpayments, and 
improve the timeliness of providing CRF information to municipalities. 


The Ministry responded to our recommendations with commitments to either take 
corrective action or to consider our recommendations in its current review of the CRF 
program. 


CURRENT STATUS OF RECOMMENDATIONS 


The Ministry of Finance advised us that it has considered all of our recommendations from 
the 2001 Annual Report. The Ministry’s response to our recommendations has resulted in 
administrative improvements to certain components of the program, as well as a 
commitment to an annual update of the active program data that determine funding 
eligibility. The Ministry also advised us that it had responded to some recommendations by 
seeking and obtaining confirmation from the government of its policy framework for the 
CRE The current status of each of our recommendations is outlined below. 


REVENUE NEUTRALITY 


Recommendation 


To ensure that future municipal financial support continues to meet the governments overall 
municipal support objectives, the Ministry should work with the Ministry of Municipal Affairs 
and Housing and incorporate in its approach an assessment of: 


¢ changes in local service delivery needs; and 


© current municipal taxing capacity. 
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If, as a result of the Community Reinvestment Fund (CRF) review, the Ministry decides to 
continue with the CRF approach to municipal funding, it should develop performance 
indicators to measure its achievement of revenue neutrality on an ongoing basis. To be able to 
assess the achievement of the objective of revenue neutrality and issues of divergent impacts of the 


CRE formula, the Ministry should ensure that its review considers: 
© =the extent to which the CRF reflects actual Local Services Realignment costs incurred; 


° the reconciliation of forecasted costs to actual costs at each year-end and subsequent 
payment adjustments; 


° the distribution of any required savings efficiencies across the province based on analysis 
and empirical information; and 


° the implications of up-to-date current value assessment data in its determination of 
municipal tax room. 


Current Status 


In 2001, prior to the release of our report, the Ministry completed a review of the CRF 
program. As part of the review, it consulted with municipalities to determine whether to 
continue with the existing CRF funding model or to adopt an alternative model. Based on 
the input received, the government decided to maintain the CRF with some administrative 
improvements. 


We understand that the continuation of the CRF approach to municipal funding has been 
reaffirmed by the government. Notwithstanding our previous concerns, the Ministry 
maintains its position that revenue neutrality has been achieved through the operation of 
the formula guiding the CRF Base calculation—namely, that municipalities whose LSR 
costs exceed the residential tax revenues granted to them in 1998 are eligible for a CRF 
Base Grant. The funding allocations arrived at through this CRF Base calculation reflect 
government policy and represent, by the Ministry’s definition, the performance indicator of 
revenue neutrality. 


With respect to our concern that the costs for fully transferred programs (that is, programs 
that are no longer administered by the province and are not cost shared) were frozen at the 
level existing at the point in time the program transferred, the Ministry indicated that this 
continued to be the case for the majority of fully transferred programs, including: Children’s 
Aid Societies; Public Housing; Airports; Septic Inspections; Gross Tax Receipts; and 
Property Assessment. However, the Ministry informed us that the fully transferred Farm Tax 
Rebate and Managed Forests/Conservation Land Rebate programs have now been 
reclassified as “active” programs. Accordingly, LSR costs associated with these programs will 
continue to be updated annually. Furthermore, since our audit, the province has taken back 
full responsibility for both the operating and capital costs related to GO Transit. In addition, 
it has assumed one-third of the capital funding costs related to Municipal Transit (operating 


costs, however, continue to be frozen). 


eee 
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With respect to our concern that the current value assessments (CVAs) on which 
municipalities based their property taxes, and hence on which the “tax-room” allocations 
used by municipalities to help fund LSR costs were based, had not been updated by the 
most recent CVA changes, the government has decided that it will not update the tax-room 
allocations to reflect the most recent CVA changes. However, the Ministry has advised us 
that LSR costs for the Farm Tax Rebate and Managed Forests/Conservation Land Rebate 
programs will continue to be updated to reflect the most recent CVA data. 


A third concern in our 2001 Annual Report was that the government required 
municipalities to achieve varying amounts of savings (depending on municipality size) as 
part of its calculation of CRF entitlements—without ensuring that the savings targets were 
supported by empirical or analytical evidence. The Ministry advised us that the government 
has reviewed and confirmed that the original CRF formula, which incorporates municipal 
savings targets, will continue to be used to determine municipalities’ CRF Base allocations. 
Accordingly, the required municipal savings targets have remained unchanged since 1998. 


CRF BONUS AND SUPPLEMENTARY ASSISTANCE 


Recommendation 


The Ministry should conduct regular reviews of the bonus and supplementary-assistance 
components of the Community Reinvestment Fund to ensure that they are achieving the 
government s objectives. 


Current Status 
The 2002 Community Reinvestment Fund (CRF) payments included the CRF bonus and 


supplementary-assistance components. In addition, the government paid a Transit Bonus to 
offset any reduction in CRF entitlements resulting from lower municipal transit capital and 
GO Transit costs when the province took back responsibility for the Municipal Transit and 
GO Transit programs. The Ministry advised us that the government has reviewed and 
approved the continuation of these components of the CRF and accordingly, funding 
allocations reflect that policy approval. 


PROGRAM ADMINISTRATION 


Monitoring of Municipalities 


Recommendation 
Tf the Community Reinvestment Fund (CRF) continues in its current form, the Ministry should 


determine whether municipalities are adhering to program requirements by: 


° reviewing municipal cash and working capital balances to ensure CRF funds are being 
used as intended by the government; 
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¢ following up with all municipalities that reported tax increases between 1999 and 
2000 to determine why the increases were necessary; and 


¢ assessing whether municipalities with windfall gains have passed these benefits on to 
taxpayers. 


To ensure that municipal fiscal planning 1s not negatively impacted, the Ministry should also 
work with municipalities and the Ministry of Municipal Affairs and Housing to determine if 
and when it would be appropriate to allow the allocation of CRF funds to municipal reserve 


accounts. 


Current Status 


Subsequent to our 2001 audit, the government rescinded its policy of barring 
municipalities from allocating CRF funds to local reserve accounts. The current policy is 
that the decision whether or not to use CRF funds immediately is a local one. The Ministry 
has also streamlined its reporting requirements for municipalities in 2002 and 2003 by 
eliminating areas where information requests were previously duplicated. 


The Ministry has also taken the position that monitoring the use of CRF funds by 
municipalities infringes on municipal autonomy and may restrict budgetary decisions. 
Given that municipalities are primarily accountable to local taxpayers, the Ministry’s view is 
that this accountability relationship is in itself a monitoring tool. As a result, the Ministry was 
not planning to follow up with municipalities to determine whether tax increases reported 
by municipalities were in accordance with CRF program requirements and whether 
municipalities that experience annual windfall gains as a result of Local Services 


Realignment (LSR) have used these gains for the benefit of local taxpayers. 


Overpayments 


Recommendation 


To ensure that the Community Reinvestment Fund (CRE) payments are appropriately made 
with due regard for economy, the Ministry, in its review, should consider the recovery of CRF 
overpayments or develop a strategy to minimize their occurrences. 


Current Status 


Following our 2001 audit, the government originally planned to adjust the 2003 CRF 
entitlements of those municipalities that received an overpayment in 2001. This process 
would have allowed municipalities with CRF reductions sufficient time to budget for their 
funding decrease. Municipalities were notified of this through a letter signed jointly by the 
Deputy Minister of Finance and the Deputy Minister of Municipal Affairs and Housing in 
November of 2001. . 
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However, the government made a subsequent policy decision to keep the 2003 CRF 
payments at the same level as the 2002 payments. Accordingly, there has been no recovery 
of previous years’ overpayments. 


Municipal Information Requirements 


Recommendation 


To improve municipalities’ ability to accurately project provincial funding when they set 
municipal tax rates and to facilitate the accurate reporting of such funding in municipal year- 
end financial statements, the Ministry should work to improve the timeliness of the information 
it provides to municipalities regarding Community Reinvestment Fund entitlements. 


Current Status 


The Ministry has committed to informing municipalities of the amount of their 
Community Reinvestment Fund (CRF) entitlements before the beginning of each 
municipal fiscal year. Accordingly, CRF allocations are now to be announced each October. 
In the fall of 2001, municipalities were advised of their 2002 CRF allocation. Similarly, the 
2003 allocations were announced in the fall of 2002. 


The Ministry advised us that it has also improved the transparency of CRF data. Previously, 
the year-end reconciliation data for some programs were based on interim estimates of 
program costs. The administrative improvements introduced in 2001 now ensure that CRF 
payments are reconciled to actual, final LSR program costs for active programs for all 
municipalities. 
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MINISTRY OF FINANCE 


4.08—Gasoline, Fuel, and Tobacco Taxes 
(Follow-up to VFM Section 3.08, 2001 Annual Report) 





BACKGROUND 


For the 2002/03 fiscal year, the Ministry collected commodity taxes on gasoline, fuel, and 
tobacco that totalled $4.2 billion. For the 2000/01 fiscal year, the amount of taxes collected 
totalled $3.25 billion, which represented approximately 6.7% of the province’s total 


taxation revenue from all sources. 


We concluded in 2001 that the Ministry’s policies, procedures, and technology systems did 
not provide the information necessary to ensure that all gasoline, fuel, and tobacco taxes due 
were being declared and paid in accordance with statutory requirements. 


With respect to the collection of gasoline and fuel taxes, we found that the Ministry did not: 


* obtain information on the amounts of gasoline and diesel fuel produced in Ontario and 
reconcile those amounts to reported sales to ensure that tax was being paid on all 
gasoline and fuel production except for legitimate tax-exempt sales; 


¢ regularly compare the billions of dollars of reported tax-exempt sales and purchases 
between collectors to ensure that large discrepancies were adequately resolved or 
assessed for tax; and 


* verify the completeness and accuracy of reported imports and exports by comparing 
them to independent information provided by inter-jurisdictional transporters, 
including pipelines. 


With respect to the collection of tobacco taxes, we found that: 


* information on the quantity of cigarettes produced in Ontario was not obtained and 
compared to the quantity of reported sales to ensure that tax was being paid on all 
cigarette production except for legitimate tax-exempt sales; 


* amore effective system for marking tax-paid cigarettes needed to be implemented; 


* the completeness and accuracy of reported tobacco imports and exports was not verified 
with independent information such as that provided by inter-jurisdictional transporters 
of tobacco products; 


* there was no assurance that the tax on tobacco imports by eerie importers was 


being declared and paid; and 


* the Ministry needed to consider the need for developing an allocation system for the sale 
of tax-exempt cigars on native reserves similar to the one in place for cigarettes. 
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At the time of our audit, we had made recommendations for improvements in each of these 
areas and received commitments from the Ministry that it would take the necessary 
corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


In June 2003, the Ministry advised us of the current status of the actions taken to address 
each of our recommendations. We are pleased to note that in many areas, substantial 
progress appears to have been made in implementing the necessary corrective action, as 


detailed in the following. 


GASOLINE AND DIESEL FUEL TAXES 


Gasoline and Diesel Fuel Production 


Recommendation 


To help ensure that all gasoline and diesel fuel produced in Ontario 1s accounted for as either 
taxable or tax-exempt sales, the Ministry should require refiners to submit information on 
gasoline and diesel fuel produced, compare it to reported taxable and tax-exempt sales, and 
periodically verify the accuracy of the production information received. 


Current Status 


The Ministry revisited the legal implications of this recommendation and determined that 
legislative authority was necessary to require refiners to submit information on production. 
As a result, new registration requirements for manufacturers are now in place in Section 2.1 
of the Fuel Tax Act and Section 2.1 of the Gasoline Tax Act and were effective as of 

July 1, 2003. A bulletin outlining the registration requirements was published in 

February 2003. Another bulletin outlining the specific reporting requirements will be 
issued upon the filing of regulations establishing the reporting details. 


It is estimated that implementation of the reporting requirements will take approximately 
12 months from when the requirements are established if'a manual system for registration 
and reporting is established, or 24 months if a more effective electronic system is developed. 
Funding for and availability of information technology resources may impact on these 
estimated delivery time frames. 


Gasoline and Diesel Fuel Tax Return Processing— 
Tax-exempt Sales and Purchases Between Collectors 


Recommendation 


To help ensure that the correct amounts of gasoline and diesel fuel taxes are declared and paid, 
the Ministry should: 
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° verify the accuracy of reported tax-exempt sales and purchases between collectors and 
ensure that large discrepancies are identified, followed up, and assessed where 
warranted; and 


¢ clearly communicate what constitutes a sale. 


Current Status 

The Ministry has identified stronger accountability and reporting mechanisms and 
discussed them with industry stakeholders. These discussions revealed practical problems 
with respect to reporting sales information, including the need for an identifier on “swap” 
contracts (wherein collectors trade petroleum products on an in-kind basis) so that the 
information in the contracts can appear on invoices. 


The Ministry has prepared a draft bulletin that addresses concerns raised by industry 
stakeholders in the course of the discussions noted above. The Ministry is expected to issue 
the bulletin before the end of 2003, pending further industry consultations and resolution 
of technical challenges to the implementation of reporting requirements. 


Gasoline and Diesel Fuel Tax Return Processing— 
Gasoline and Diesel Fuel Imports and Exports 


Recommendation 


To help ensure that all imports and exports of gasoline and diesel fuel are accounted for and that 
the correct amount of tax is declared and paid, the Ministry should: 


° ensure that all gasoline and diesel fuel tax collectors and inter-jurisdictional 
transporters, including pipelines, submit the information required of them by law; 


° verify the accuracy of reported imports and exports by comparing them to information 
provided by inter-jurisdictional transporters—when significant variances are 
identified, they should be investigated and resolved on a timely basis; 


° provide the Canada Customs and Revenue Agency with an up-to-date list of registered 
importers to ensure that unregistered importers pay the applicable tax at the time of 
import; and 


° develop the appropriate tax return forms for inter-jurisdictional pipeline transporters. 


Current Status 


The Ministry informed us that reporting requirements for collectors and inter-jurisdictional 
transporters, including pipelines, with respect to imports and exports have been brought to 
the attention of non-compliant registrants. A bulletin outlining requirements for 
transporters, including pipelines, was published in April 2003. In addition, Bill 198 


established more effective sanctions for failure to file sales and transporter returns. 


318 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





To facilitate the verification of reported information, the Ministry was considering, at the 
time of our follow-up, electronic service-delivery options under the electronic reporting 
initiative of the Canadian Fuel Tax Project (a project mandated to simplify and standardize 
the administration of fuel taxes across all Canadian jurisdictions). 


An updated border collection agreement for petroleum tax collection that includes a 
provision for regular information exchanges between the Canada Customs and Revenue 
Agency (CCRA) and the Ministry has been reviewed by the Ministry’s Legal Services 
Branch. Finalization of the agreement with the CCRA is expected by the end of the 
2003/04 fiscal year. The agreement should ensure that the CCRA has up-to-date 
information on registered importers to enable the collection of taxes from unregistered 
importers. 


‘Tax return forms and schedules for inter-jurisdictional pipeline transporters have been 
finalized and are currently in use. 


Gasoline Tax Refunds 


Recommendation 


To ensure that only eligible gasoline sales are exempted from tax, the Ministry should review 
refund vouchers submitted by collectors for completeness and reasonableness of exemptions claimed 
based on assessed risks and follow up on questionable or incomplete vouchers to determine 
whether or not the purchases qualify as tax-exempt. 


Current Status 

The Ministry informed us that the review of the refund claims in collector returns using 
risk-based sampling techniques had been suspended during our audit in 2001 while 
resources were redirected to other activities. The Ministry advised us that this review was 
resumed in June 2001 and that refund claims continue to be monitored. As a result, eight 
warning notices had been issued to non-compliant retailers; those retailers are now required 
to file their returns and supporting vouchers directly with the Ministry. 


The Ministry also informed us that the vouchers submitted by retailers that previously 
submitted questionable claims are now closely scrutinized prior to authorizing any refund. 


Fuel Acquisition Permits 


Recommendation 


To ensure that tax-exempt diesel fuel purchased by holders of Fuel Acquisition Permits is used only 
for eligible purposes, the Ministry should: 


° monitor and assess the reasonableness of the tax-exempt diesel fuel purchases reported by 
Fuel Acquisition Permit holders; and 
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° conduct periodic audits or inspections of all types of permit holder facilities, based on 
the assessed risks. 


Current Status 

The Ministry informed us that it now annually reviews renewals of Fuel Acquisition Permits 
used to buy tax-exempt diesel fuel for kerosene repackaging, and field inspections of permit 
holder facilities are conducted where actual business practices could not be confirmed as 
appropriate. 

The Ministry also informed us that it reviewed, prior to the 2002 renewal cycle, all of the 
163 accounts of permit holders buying the tax-exempt fuel for tobacco curing. Only 86 
permits were renewed. Inspectors visited approximately 30 of the 86 tobacco curers granted 
renewals and found that the curing process had changed such that the curing process no 
longer required clear diesel fuel. The Ministry has since cancelled all remaining permits for 
tobacco curers. 


The Ministry is also participating in the exempt sales sub-project of the Canadian Fuel Tax 


Project to determine alternative methods of administering tax-exempt sales. 


TOBACCO TAXES 


Tobacco Tax Return Processing—Cigarette Production 
and Control 


Recommendation 
The Ministry should ensure that all cigarette manufacturers: 


© submit the required schedules indicating the number of Ontario-marked cigarettes 


produced, shipped, and on hand; and 
* provide information about the amount of yellow tear-tape acquired and used. 


The Ministry should then use that information to assess the reasonableness of reported taxable 
sales, 


Current Status 


The Ministry informed us that all registered manufacturers were notified of the mandatory 
requirement to file the appropriate inventory distribution and sales schedules with their 
monthly returns and are now doing so on a regular basis. Upon receipt of the returns, 
Ministry staff routinely reconcile the two schedules and follow up on any discrepancies. 


Amendments to the 7obacco Tax Act require tear-tape manufacturers to account for the 
manufacture and sale of tear-tape and to limit sales to registered manufacturers. The 
amendments also require that cigarette manufacturers purchase tear-tape only from 
manufacturers that have been issued a manufacturing permit by the Ministry and that every 


_ ESS 
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permit holder “account for and take reasonable steps to safeguard tear-tape in its 
possession.” However, at the time of our follow-up, the amendments had not yet been 
proclaimed and were therefore not in force. 


Tobacco Tax Return Processing—Tobacco Imports and 
Exports 


Recommendation 


To ensure that tobacco product import and export reports are complete and accurate and that 
the correct amount of tax has been declared and paid, the Ministry should: 


° obtain the required information from all inter-jurisdictional transporters or, if that 
information is unavailable, obtain other evidence, such as customs declarations, to 
assess the completeness and accuracy of reported imports and exports; and 


° regularly obtain information from the Canada Customs and Revenue Agency detailing 
tobacco-product imports by unregistered importers and use it to determine whether the 
correct amounts of tobacco tax have been declared and paid. 


Current Status 


‘Transporters carrying tobacco are now required to register with the Ministry and report the 
names of the exporters and importers for which they are carrying tobacco. Failure to do so is 
being treated as a non-compliance issue, and the Ministry informed us that it has contacted 
specific transporters to address deficiencies. The Ministry was expecting that, by September 
2003, it would be reviewing monthly returns submitted by transporters and, where 
necessary, taking appropriate follow-up action. In that regard, a regulation detailing the 
requirements for inter-jurisdictional tobacco transporter returns had been drafted and 
reviewed, and approval was pending. Also, a bulletin outlining inter-jurisdictional tobacco 
transporter requirements had been drafted and approved. 


An updated report on tobacco product imports was received from the CCRA in late 2002, 
after a protracted delay due to legal and policy issues. The report has been analyzed and 
integrated into the Ministry’s audit plan for the 2003/04 fiscal year. In addition, after the 
border collection agreement for petroleum tax collection (see “Current Status” of “Gasoline 
and Diesel Fuel Tax Return Processing—Gasoline and Diesel Fuel Imports and Exports”) is 
finalized, the Ministry is to commence negotiations with the CCRA to enhance the border 


collection agreement for tobacco tax collection. 
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Cigar Taxes 


Recommendation 


To ensure that the quantity of tax-exempt cigars sold on native reserves is reasonable, the 
Ministry should consider the need for an allocation system for cigars similar to the one in place 
for the sale of tax-exempt cigarettes, and if considered advisable, initiate its development. 


Current Status 
The Ministry informed us that it has completed a study of the advisability of an allocation 


system for cigars; the study was being reviewed at the time of our follow-up. 


SECURITY REQUIREMENTS 


Recommendation 
To help protect its financial interests, the Ministry should: 


° consider whether its best interest would be served by requiring security from all 
registrants; and 


° ensure that all registrants required to post security do so in the stipulated amount. 


Current Status 


The Ministry informed us that it was reviewing the grandfathering concession whereby 
registrants in operation before 1992 were exempted from security requirements. The review 
was expected to be completed this year. In this regard, we noted that three collectors lost 
their exempted status due to amalgamations and were requested to post financial security. 


The Ministry also informed us that it is to annually review the required amount of security 
for all collector accounts through desk audits. In addition, linking accounts in the Ministry’s 
Motor Fuels and Tobacco Tax Branch to the Ministry’s core tax systems, such as [TAS 
(Integrated Tax Administration System) and BASYS (the Ministry’s commodity sales tax 
system), should help the Ministry identify changes in the pertinent entities (manufacturers, 
collectors, and importers) on a timely basis. 


GASOLINE, DIESEL FUEL, AND TOBACCO TAX AUDITS 


Recommendation 


To ensure that audit work is completed satisfactorily and clearly determines whether or not the 
correct amount of gasoline, diesel fuel, and tobacco tax has been declared and paid, the Ministry 
should: 


° develop detailed audit programs that comply with the work requirements of the 
Ministry's own Audit Procedures Manual for inclusion in each audit file; 
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° encourage supervisory input at the planning stage to ensure high-risk areas are 


appropriately addressed; 


° ensure that the required work is completed satisfactorily and that any unresolved 
variances are assessed when warranted to prevent the results of further audit work from 
becoming statute barred; and 


¢ complete audits on a three- to four-year cycle to ensure that returns do not become 
statute barred. 


Current Status 


The Ministry informed us that revised audit programs are in place and being tested in the 
field. According to the Ministry, these programs provide greater detail as regards work 


requirements and facilitate supervisory input and review of completed work. 


Field and desk audit plans are being co-ordinated to ensure audits are completed prior to 
statute barring of potentially material tax liabilities. 


FIELD INSPECTIONS 


Recommendation 


To maximize the benefits of its inspection program for encouraging compliance with gasoline, 
diesel fuel, and tobacco tax requirements, the Ministry should: 


° assess whether the number of inspectors is sufficient to promote voluntary compliance on 
a broad scale; 


° base its performance targets for the type, number, and location of inspections to be 
undertaken on the periodic assessment of known risks of tax evasion schemes; and 


° monitor the results of inspections by type, number, and location, and periodically make 
any changes needed to ensure that its resources remain effectively deployed. 


Current Status 


The Ministry has recognized that more inspectors are needed to provide an appropriate 
level of province-wide coverage. An increase in inspection resources has been proposed, but 
the Ministry advised us that the need for additional inspections must compete for limited 
resources with the needs of other ministry enforcement and audit functions. 


A computer-based planning and management information system is being investigated for 
use in the Motor Fuel and Tobacco Tax Branch’s field inspection unit. A results-reporting 
feature will be included for better monitoring of performance. In the meantime, manual 
forms are being used for performance monitoring, with spreadsheet software being used to 
control and analyze results reporting. 
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BUSINESS PROCESS RE-ENGINEERING 


Recommendation 


Given that many of our preceding recommendations could be implemented most efficiently and 
effectively through the use of information technology, the Ministry should give priority to 


completing its business re-engineering project as soon as possible. 


Current Status 


The Ministry advised us that the business re-engineering project was implemented in 
November 2002. Additional functionalities are being added to accommodate budget and 
legislative changes and new business requirements. 


EFFECTIVENESS MEASURES 


Recommendation 


To determine the extent to which it is meeting its objectives of encouraging voluntary compliance 
with and deterring the evasion of gasoline, diesel fuel, and tobacco taxes, the Ministry should 
develop and implement the necessary performance indicators. 


Current Status 


The Ministry informed us that performance indicators resulting from audits are being 
statistically summarized and analyzed for assessment amounts and trends. At the time of our 
follow-up, a similar system for inspections was being developed. 


ESSE 
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MINISTRY OF HEALTH AND LONG-TERM CARE 


4.09—Drug Programs Activity 
(Follow-up to VFM Section 3.09, 2001 Annual Report) 





BACKGROUND 
The Drug Programs Branch of the Ministry of Health and Long-Term Care is responsible 


for co-ordinating the provision of prescription drugs and related products to eligible 
Ontarians. To this end, the Ministry administers transfer payments provided by the Ontario 
Drug Programs Activity for the Ontario Drug Benefit Program, the Trillium Drug Program, 
and the Special Drugs Program. Legislative authority for the Ontario drug programs 
transfer payments are established under the Ontario Drug Benefit Act, the Drug 
Interchangeability and Dispensing Fee Act, and the Health Insurance Act. 


For the 2002/03 fiscal year, the programs’ expenditures were $2.6 billion, of which 

$505 million was recovered from the Ministry of Community, Family and Children’s 
Services for drug benefits paid for social-assistance recipients. For the 2000/01 fiscal year, 
the programs had total expenditures of $1.98 billion, of which $413 million was recovered 
from the then Ministry of Community and Social Services for drug benefits paid for social- 


assistance recipients. 


At the time of our 2001 audit, although we noted that the Ministry had introduced a 
number of initiatives to manage drug expenditures, we found that the Ministry had not 
given sufficient consideration to the prices it was paying for drugs. Specifically, we found 


that: 
¢ Delays in adding approved generic drugs to the Ontario Drug Benefit Formulary and 


in implementing manufacturers’ price reductions resulted in lost savings totalling 
$17 million over a two-year period. 


¢ The Ministry had not reviewed the effectiveness of the generic pricing practices or 
routinely compared the prices it was paying for drugs with the prices paid by other 
jurisdictions. In this regard: 
- Forasample of generic products we tested, the Ministry would have saved 


approximately $54 million annually had it paid the same prices as Saskatchewan for 
these products. 


- Another jurisdiction, using a competitive acquisition process, was able to obtain 
prices for certain drugs that were, on average, 60% lower than those obtained by 
Ontario. Annual potential savings to the Ministry could have been as much as 
$140 million if it had been able to obtain the same prices for these drugs. 
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The Ministry generally had adequate procedures in place to ensure compliance with 
legislation and that claims were properly approved, processed, and paid. However, we noted 
that: 


¢ The Ministry had not substantiated whether as many as 180,000 of the recipients who 
were granted temporary eligibility for the Ontario Drug Benefit Program in 
1999/2000 were in fact entitled to benefits. 


¢ The Ministry forgave $1.5 million to be recovered from pharmacies resulting from a 
1997 verification of claims for limited-use drugs. 


¢ The Ministry needed to improve the procedures for paying Special Drugs Program 
invoices—we found that, for one drug we selected for audit, the Ministry had been 
overcharged $475,000 over a five-year period (the Ministry was in the process of 


recovering the overpayment from the manufacturer). 


We made a number of recommendations for improvement and received commitments 
from the Ministry that it would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


According to information obtained from the Ministry of Health and Long-Term Care, the 
Ministry has taken some action on most of the recommendations we made in our 
2001 Annual Report. The current status of our recommendations is as follows. 


ONTARIO DRUG BENEFIT AND TRILLIUM DRUG 
PROGRAMS 


Drug Use Review 


Recommendation 


To help ensure that Ontario s drug programs encourage the economic and appropriate use of 
prescription drugs and result in optimal improvement in the health status of recipients, the 
Ministry, in consultation with other stakeholders, should: 


¢ establish a drug use review program; and 


° ensure that the Health Network System provides accurate and complete information to 
implement drug use review. 


Current Status 
The Ministry advised us that: 
¢ The Ministry “has established the Drug Strategy Review (DSR) with a mandate to find 


ways to optimize pharmaceutical care in order to ensure access to the drugs Ontarians 


need, now and in the future. The DSR is undertaking a review of the Ontario Drug 
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Benefit Program and developing a strategy aimed at improving pharmaceutical care for 
patients that will include an examination of access to new and existing drugs, cost- 
effectiveness and pricing of drugs, appropriate drug use, and program administration.” 


¢ A ministry-funded committee, the Ontario Program for Optimal Therapeutics, was 
focusing on assessing the utilization of drug-use prescribing guidelines. ‘Iwo areas for 
review (statins and opiod narcotics) had been identified. The Ministry was reviewing the 
process for conducting the review of these two areas with the pharmaceutical industry 
and the Drug Utilization Advisory Committee to ensure that it was aligned with other 
ongoing work, such as the Drug Benefit Formulary modernization undertaken by the 
Drug Quality and Therapeutics Committee (DQTC). 


¢ The Ministry was using work conducted by the Institute for Clinical Evaluative Sciences 
in a number of therapeutic areas to support ongoing work with DQTC’s Formulary 
modernization and to identify areas that may benefit from additional measures to 
improve prescribing and utilization. 


The Ministry informed us that it had contacted the College of Physicians and Surgeons of 
Ontario and was in ongoing discussion with the College to generate a list of physician 
identification numbers for the Health Network System database to ensure that prescribers 





can be identified. In addition, pharmacies have been instructed that valid identification 
numbers must be used unless the situation is exceptional. 


The Drug Formulary—Timely Updates to the Formulary 


Recommendation 


To help maximize potential savings to the Drug Programs Activity, the Ministry should pursue 
more timely updating of the Ontario Drug Benefit Formulary when: 


¢ adding approved generic drugs; and 


° implementing manufacturers’ price reductions. 


Current Status 

The Ministry indicated that ministry staff had met with Health Canada staff to further 
harmonize the Ministry’s drug-listing process for generic products with that of Health 
Canada. The Ministry also participated in a federal/provincial/territorial discussion on 
streamlining generic products in March 2003 and is currently working on streamlining 
proposals to ensure timely updating of the Formulary. 


According to the Ministry, the Ministry will be considering options for decreasing the time 
for listing streamlined generic products, which would also enable the Ministry to take 


timelier advantage of manufacturers’ price reductions. 
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The Drug Formulary—Formulary Modernization 


Recommendation 


The Ministry should ensure that drugs listed in the Ontario Drug Benefit Formulary are 
regularly reviewed so that the Ontario Drug Benefit Program only covers the cost of drugs that are 
appropriate and cost effective. 


Current Status 
The Ministry advised us that, in August 2002, the Drug Quality and Therapeutics 


Committee discussed the review of a number of therapeutic categories of drugs and that six 
drug category reviews—for example, reviews of drugs for osteoporosis and multiple 
sclerosis—had been completed since December 2002. Three additional reviews had been 
initiated and were to be completed early in the 2003/04 fiscal year. 


Pricing 


Recommendation 


1o better control the drug costs of Ontario’s drug programs and to enable the Ministry to more 
effectively negotiate prices with drug manufacturers, the Ministry should routinely compare the 
prices it pays for drugs with the prices paid by other provinces. 


The Ministry should also review the generic pricing rule to ensure that it does not impede the 
Ministry from obtaining generic drugs at the lowest possible price. 


Current Status 
According to the Ministry, Ontario is a member of the Federal/Provincial/Territorial 


Working Group on Drug Prices, which is examining price initiatives across jurisdictions. As 
part of its examination of pricing of drugs, the Drug Strategy Review is to examine medium- 
and long-term pricing options for both brand-name and generic drugs to achieve fair and 
reasonable prices for new and existing drugs. 


The Ministry advised us that it was reviewing the present pricing policy for generic drugs. 
Pricing—Pricing Options 


Recommendation 


1o help ensure that it obtains better value for money for its drug expenditures, the Ministry 
should assess the costs/benefits of pricing options that have been successfully implemented in other 


Jurisdictions. 


a ee 
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Current Status 


As mentioned earlier, the Ministry is working with other Canadian jurisdictions on this 
issue. As part of this work, the Ministry is to be undertaking research on key pharmaceutical 
policy trends, which would include a review of pricing strategies in select jurisdictions. 


Written Agreements with Brand-name Drug 
Manufacturers 


Recommendation 
To help ensure that drug costs are more effectively managed, the Ministry should: 


¢ evaluate the extent to which the current written agreement process with drug 
manufacturers is meeting its objectives; and 


° make improvements as required. 


Current Status 

According to the Ministry, an internal report on the written agreement process had been 
completed at the time of our follow-up, and a formal review of the written agreement 
process is to be undertaken by the Drug Programs Branch. The Ministry indicated that a 
business case for the formal review had yet to be prepared to secure resources and to hire an 
external consultant. 


Health Network System 


Recommendation 
When selecting a vendor to provide long-term services without using a competitive process, the 
Ministry should ensure that it: 


° receives value for money through respective contracts with such vendors; and 


° complies with Management Board of Cabinet directives. 


Current Status 


At the time of our 2001 audit, a vendor had been selected without using a competitive 
process to provide long-term services in developing and maintaining the Health Network 
System. The Ministry advised us that a consultant had recently prepared a business case and 
a request for resources (RFR) for a vendor of record to conduct a value-for-money audit of 
the current contractual arrangements with this long-term-service provider. The business case 
and RFR were in the process of receiving approval from ministry senior management. 
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Health Network System—Claims Processing 


Recommendation 


To help ensure that pharmacy data within the Health Network System (Network) is complete 
and accurate, the Ministry should periodically verify pharmacy registrations with the Ontario 
College of Pharmacists. 


To help ensure that only eligible individuals receive benefits through the Ontario Drug Benefit 
Program, the Ministry should: 


° review and follow up on exception reports, which identify mismatched or missing 
information in the Network's recipients database; and 


° regularly compare data in the Ministry of Community and Social Services’ database 
with the Network's database. 


Current Status 


The Ministry advised us at the time of our follow-up that no changes had been made to the 
verification process and that the Ministry would continue to review the verification process 
to identify opportunities to address the first part of our recommendation. 


The Ministry also advised us that it was continuing to conduct periodic reviews of exception 
reporting and was working with the Ministry of Community, Family and Children’s Services 
(the former Ministry of Community and Social Services) to ensure recipient information is 
as accurate and up to date as possible. 


TEMPORARY ELIGIBILITY 


Recommendation 


To help ensure that temporary eligibility is being granted only where justified, the Ministry 
should: 


° periodically verify the adequacy of supporting documentation maintained by 
pharmacies where there are significant numbers of unsubstantiated claims; and 


° together with the Ministry of Community and Social Services (MCSS), expedite 
necessary improvements to the MCSS database. 


Current Status 


The Ministry advised us that the Ministry of Community, Family and Children’s Services 
(the former Ministry of Community and Social Services) had moved from a weekly to a 
daily feed to update eligibility and personal information contained on the Health Network 
System. The Ministry indicated that the daily updates would be monitored to determine 
whether continuing them is operationally sustainable. The Ministry also indicated that it 
would review the number of temporary eligibility claims in the 2002/03 fiscal year and 





330 2003 Annual Report of the Office of the Provincial Auditor of Ontario 


compare it to the numbers in previous years. Further steps would depend on the results of 
this review. 


WARNING AND INFORMATION MESSAGES 


Recommendation 


To improve the effectiveness of the Health Network Systems warnings and information messages, 
the Ministry should assess whether: 


° the existing warning and information messages need to be revised; and 


° other potential drug therapy problems, such as a therapeutic duplication check, should be 
added to the Network. 


Current Status 
According to the Ministry, staff from the Drug Programs Branch will be conducting a 


preliminary investigation of additional warning and information message modules that are 
available and will consider including these modules in the next request for proposals with 
the data supplier. The Branch also intends to investigate the feasibility and benefits of 
adding a therapeutic duplication check to the Network. 


Health Network System—System Security 


Recommendation 


To help safeguard information in the Health Network System against unauthorized use, 
disclosure, modification, damage, or loss, the Ministry should: 


¢ assign the responsibility for the Networks security to an appropriate senior manager; 
° ensure appropriate security policies and procedures are in place; 
° review staff duties to ensure that system access is appropriate; 


¢ implement more rigorous controls over the access administration process and system 
protection; and 


¢ ensure that the Network’ security is actively monitored. 


Current Status 
According to the Ministry, at the time of our follow-up: 
¢ The security administration process had been reviewed and is now managed through 


one branch manager. Notification of staff changes had been included in the security 
review. 


¢ The Drug Programs Branch was working with the Health Network System service 
provider to consolidate all security-related documentation and to update the 
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documentation where appropriate. A new security document had been finalized and 


approved. 


¢ The replacement date for the current communications network had been re-scheduled 
from April 2003 to April 2004. The Ministry indicated that it would continue to 


implement security procedures as required. 


Contract Management 


Recommendation 


To enhance accountability, the Ministry should ensure that it has adequate policies and 
procedures in place to monitor whether contracted services are carried out in accordance with 
the terms, conditions, and performance standards set out in contracts. 


Current Status 
According to the Ministry, staff from the Ministry’s internal audit branch have developed 


procedures for periodic reviews to address this recommendation. Drug Programs Branch 
staff met with internal audit staff in February 2003 to review the procedures. 


Inspections and Verification—Inspection Resources 


Recommendation 
To help ensure that inspection resources are used efficiently and effectively, the Ministry should: 


° implement needed improvements to the Pharmaceutical Audit System to facilitate the 
work of inspectors; 


° ensure inspection plans are prepared and approved by branch management; 
° provide for sufficient management review of the work of inspectors; and 


° review the adequacy of the policies and procedures in the drafi manual. 


Current Status 

At the time of our follow-up, the Ministry advised us that: 

* Due to the public service strike in 2002 and the implementation of projects that had a 
higher priority than the Pharmaceutical Audit System, implementation of Phase II of 


the system had been deferred until January 2003. Phase IT has now been implemented, 
and a training program for appropriate staff has now been developed. 


¢ Inspection plans were in place. 


* The policies and procedures in the draft manual, which incorporate management 
review of the work of inspectors, have been reviewed, finalized, and put in place. 
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Inspections and Verification—Inspection Coverage 


Recommendation 


To minimize the risk of paying for invalid claims, the Ministry should ensure that sufficient 
resources are assigned for the inspection of pharmacies. 


Current Status 


The Ministry advised us that the review of resources in place to inspect pharmacies had yet 
to be undertaken. The Drug Programs Branch was working with the Ministry’s Fraud 


Programs Branch to establish the terms of reference and timeline for the review. 


Inspections and Verification—Verification Letters 


Recommendation 


To help ensure that the drug programs pay only for valid prescription claims submitted by 
pharmacists, the Ministry should implement adequate procedures to verify claims with recipients. 


Current Status 


The Ministry advised us at the time of our follow-up that a review of audit activities in other 
Be P 

provinces was being planned. When the review is complete, the Ministry intends to 

determine which additional procedures are to be implemented in Ontario. 


Inspections and Verification—Verification of Limited-Use 
Drug Forms 


Recommendation 


To help ensure that the costs of limited-use drugs are only covered where warranted, the Ministry 
should: 


© ensure that adequate procedures are in place to periodically verify that limited-use 
claims are supported by valid documentation; and 


° enforce recoveries where pharmacists do not provide adequate evidence that limited-use 
drug criteria have been met. 


Current Status 

The Ministry indicated that it was still planning to carry out audits of limited-use claims in 
the future and would make recoveries where claims were not supported by valid 
documentation. As required, the Ministry obtained Order-in-Council approval to forego 
the recovery of the outstanding funds from pharmacists related to the limited-use audit 
carried out in 1997, pursuant to subsection 5(1) of the Financial Administration Act. 
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TRILLIUM DRUG PROGRAM 


Recipient Deductibles 


Recommendation 


To better ensure that Trillium Drug Program benefits are provided in accordance with the 
intent of the Program, the Ministry should develop policies and procedures to: 


° reduce or eliminate underpayments of the deductible; and 


° recover any underpaid deductibles. 


Current Status 

According to the Ministry, in order to address the possibility of recipients receiving program 
benefits in one quarter while underpaying their deductibles in other quarters, the Health 
Network System (HNS) no longer allows pharmacists to dispense more than 30 days of 
supply of a drug beyond the end of the third quarter of the benefit year (May 30). The 
HNS now also does not allow a pharmacist to override this control if a client’s claim occurs 
during the third quarter of the year and may result in an underpayment of the deductible. 
The Ministry is examining options for the recovery of underpaid deductibles. 


The Ministry also indicated that the Drug Programs Branch had met with the Canada 
Customs and Revenue Agency (CCRA) to explore an electronic feed of income data from 
CCRA. If an electronic feed can eventually be put in place, the Branch will be able to check 
the seniors’ data and validate Trillium data on a regular basis. 


SPECIAL DRUGS PROGRAM 


Recommendation 


The Ministry should consider whether the Special Drugs Program 1s needed in its current form 
and whether the administration of the Program could be integrated with the Ontario Drug 
Benefit and Trillium Drug programs. 


Current Status 
The Ministry advised us that, in December 2002, a discussion paper with short- and long- 
term recommendations was approved by the Minister’s office. In the short term: 


¢ New brands/formats of existing substances that treat one of the conditions covered by 
the Special Drugs Program (SDP) are to be added to the list of drugs covered by the 
SDP to provide choice of products to physicians. 


¢ The SDP is to be automated to actively manage and increase acCountability in the 
system. 
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¢ The Ministry has informed hospitals, stakeholder groups, and manufacturers of the 
changes to the SDP and the process for adding new brands/formats of existing 


substances. 


The Ministry also indicated that it would continue to consult with stakeholders as it moves 
towards the longer-term objective of integration of the Special Drugs Program with the 
Trillium Drug Program. The Ministry noted that the benefits of integrating the two 
programs include equal and convenient access to drug therapy for all patients, fair 
treatment of all manufacturers, and increased accountability for claimants (that is, patients, 
physicians, and institutions/pharmacies). However, a target date for the integration of the 
two programs had not been set. 


Payment Processing 


Recommendation 
To help ensure that payments from the Special Drugs Program are reasonable, the Ministry 
should: 


¢ establish procedures to compare invoiced amounts to prices in contracts between 
hospitals and manufacturers; 





° ensure any administrative costs being paid to a hospital are justified; and 


¢ monitor the volume of drugs paid for by the Program. 


Current Status 
According to the Ministry, at the time of our follow-up: 


¢ Procedures had been updated to ensure routine checking of all invoices from hospitals. 


¢ Automation of the Special Drugs Program (SDP) would be piloted at three hospitals in 
April 2003. Initially, the automation would involve online claims adjudication for two of 
the eight categories of products that are reimbursed under the SDP. Automation would 
then be rolled out to other SDP hospitals linked to the Ministry’s Health Network 


System over a period of three to four months. 


¢ Hospitals not linked to the Network would need to purchase software and hardware in 
order to get the linkage. Automation of claims from these hospitals would occur over a 
longer period, likely four to six months. 


The Ministry also indicated that, during the phasing-in of automation, both the manual 
invoicing and online adjudication systems for reimbursements would be in place. In 
addition, options for automating the other six categories of SDP drugs were being 
researched by category, since each category has unique issues with respect to, for example, 
distribution and monitoring programs needed to ensure the safety of therapy. 
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The Ministry also indicated that it now has a process in place to check that each invoice has 
been properly priced and that all proper documentation, including support for 
administrative costs paid to hospitals, has been submitted. 


PERFORMANCE MEASUREMENT AND REPORTING 


Recommendation 


To provide better accountability to the public and the Legislature, the Ministry should develop a 
comprehensive set of performance measures and report regularly and publicly on the performance 
of the drug benefit programs. 


Current Status 


The Drug Programs Branch has posted its annual report on the ministry Web site (at Atzp-// 
www.health.gov.on.ca:80/english/public/pub/ministry_reports/odb_report01/drug_rep.html). 
The report includes sections on: financial indicators; clinical indicators; customer service; 


and operational policy. 
The Ministry advised us that, in response to the 2003 First Ministers’ Accord on Health Care 


Renewal, the Drug Programs Branch identified 11 possible performance measures. Three of 
these measures are to be investigated further through the Pharmaceutical Care Working 
Group, which was one of the five working groups established in response to The 
Commission on the Future of Health Care in Canada (headed by Roy Romanow) and the 
First Ministers’ Health Accord. 
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MINISTRY OF HEALTH AND LONG-TERM CARE 


4.10—Assistive Devices and Home Oxygen 
Programs 


(Follow-up to VFM Section 3.10, 2001 Annual Report) 





BACKGROUND 


The Assistive Devices Program and the Home Oxygen Program are administered by the 
Operational Support Branch of the Ministry of Health and Long-Term Care. According to 
the Ministry, the objective of both programs is to “financially assist Ontario residents with 
long-term disabilities to obtain basic, competitively priced, personalized assistive devices 
appropriate for the individual’s needs and essential for independent living.” Both programs 
are funded under the Ministry of Health Act. 


During the 2002/03 fiscal year, the Ministry provided financial assistance totalling 
approximately $214 million to 193,000 individuals. The Ministry also provided 
approximately $8.5 million to transfer-payment agencies for services relating to assistive 
devices. During the 2000/01 fiscal year, the Ministry provided financial assistance totalling 
approximately $184 million to 176,000 individuals. It provided approximately $8 million 
to transfer-payment agencies for services relating to assistive devices. 


At the time of our 2001 audit, we found that the Ministry did not have adequate 
procedures to ensure that it was paying the best prices for assistive devices acquired. 


Specifically, we noted that: 


¢ Ministry-initiated independent research indicated that 41% of approved renewals for 
home oxygen met no eligibility criteria for receiving ministry-funded home oxygen. 
Reducing this number by one-half could have saved the Home Oxygen Program over 
$5 million annually. 


¢ The Ministry could have saved in the order of $3 million to $5 million annually if home 


oxygen vendors were paid the same price as vendors were paid in Alberta. 


¢ For three types of commonly purchased wheelchairs, the Ministry could have saved 
approximately $1.9 million annually if it paid the same price as Quebec paid for these 
wheelchairs. 


The Ministry generally had adequate procedures in place to ensure that claims were 
properly approved, processed, and paid but needed to improve its procedures for 
measuring and reporting on the effectiveness of the programs. 


We had made a number of recommendations for improvement and received commitments 
from the Ministry that it would take corrective action. 
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CURRENT STATUS OF RECOMMENDATIONS 


According to information obtained from the Ministry, the Ministry has taken some action 
on all of the recommendations we made in our 2001 Annual Report. The current status of 
each of our recommendations is as follows. 


MANAGEMENT OF RESOURCES 
Home Oxygen Program—Eligibility—Oximetry Tests 


Recommendation 


To help ensure that funding for home oxygen is provided only to individuals who meet the 
Ministry s eligibility criteria, the Ministry should: 


¢ assess whether blood gas tests should be used rather than oximetry tests; 


° assess whether to retest home oxygen recipients earlier than a year from the time an 
individual begins receiving home oxygen; and 


° establish clear criteria, guidelines, or definitions to address situations where individuals 





are experiencing low oxygen levels during exercise or sleep. 


Current Status 
At the time of our follow-up, the Ministry indicated that: 


¢ New applicants to the Home Oxygen Program who are 19 years of age or older must 
have a blood gas test performed to show that they need oxygen therapy. At 90 days and 
at 15 months after the use of home oxygen has started, the need for oxygen must be 
reassessed. At each stage, an oximetry test must be done to confirm the continued need 


for oxygen therapy. 
¢ The Medical Criteria Task Force, which reports to the Respiratory Services Standing 


Committee established by the Ministry, made recommendations for establishing a 
reliable, standardized endurance protocol for exercise hypoxemia. In response, the 
Home Oxygen Program was working with the Task Force to determine specific 
eligibility criteria based on this protocol and plans to establish and communicate the 


new policy to stakeholders by fall 2003. 
¢ The Medical Criteria Task Force has agreed that further investigation and research is 


required to determine the implications of nocturnal hypoxemia. If patients spend 30% 
or more of the night with reduced oxyhemoglobin saturation, this may merit funding 
for home oxygen. However, the recommendation from the Respiratory Services 
Standing Committee was that funding for nocturnal hypoxeniia should not be made 
available at this time. 
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Ostomy Grants 


Recommendation 

To ensure that payments for ostomy supplies over the maximum amount are warranted, the 
Ministry should, at a minimum, assess the current needs of individuals receiving significantly 
more funding than the current maximum grants. 


Current Status 

At the time of our follow-up, the Ministry indicated that it has contracted with an 
organization to undertake an assessment of individuals receiving significantly more funding 
than the current maximum grants. A survey tool for the assessment has been approved, and 
the Ministry expects that a summary report will be available in summer 2003. 


Replacement Devices 


Recommendation 

To better ensure that devices are only replaced when justified, the Ministry should review the 
reasonableness of its established replacement periods, particularly for those devices that are often 
being replaced early 


Current Status 

At the time of our follow-up, we were informed by ministry staff that effective March 31, 
2002 four standing committees organized for the four medical device categories— 
Prosthetics and Orthotics; Respiratory Services; Sensory; and Wheelchair, Positioning and 
Ambulation Aids—were operating. Each committee comprises representative vendors, 
manufacturers, health care professionals, and consumers, and their role is to provide policy 
advice on various issues, including eligibility. The four standing committees will also be asked 
to advise on the appropriateness of current replacement periods. 


As well, the Assistive Devices Program was establishing a project to analyze the actual 
replacement time periods for hearing aids. 


Computer Purchases 


Recommendation 


To help ensure that the Assistive Devices Program 1s paying competitive prices for computer 
equipment, the Ministry should: 


¢ pay only for approved devices; 
© review the approved amount and pay only what 1s necessary; and 


¢ ensure clients provide, at a minimum, one price quotation from the ministry-funded 
equipment pool or another supplier. 





Assistive Devices and Home Oxygen Programs 339 





Follow-up Section 4.10 


Follow-up Section 4.10 








Current Status 
At the time of our follow-up, the Ministry indicated that: 


¢ The Ministry now had, in its procedures for approving invoices, all of the necessary 
checks for ensuring that only approved computers to be used for communication aids 
or aids for the visually impaired are paid for and that approvals are given only for what is 
necessary. 


¢ The Assistive Devices Program’s Finance and Audit unit was verifying invoices submitted 
to the Program, following payments, to ensure that vendors were following program 
policies. Unit staff began site visits with selected vendors commencing in September 
2002. 


¢ The Ministry had completed a pricing framework for assistive devices, including 
computer equipment, and it had hired a consulting firm to complete a pricing review to 
ensure that prices listed for products were competitive. 


¢ The policy for communication aids and visual aids now clearly requires that clients 
submit at least two price quotations when an equipment pool or registered vendor is not 
used. 


Approval of Devices for Coverage 


Recommendation 


To help ensure that only appropriate devices are funded under the Assistive Devices Program, the 
Ministry should document procedures for the inclusion of all new devices in each category in the 
Program s catalogue. 


Current Status 


The Ministry has updated procedures for adding and removing devices from the catalogue 
and has posted them on the ministry Web site to provide clear policy information for 
stakeholders. The Ministry advised us that, at the time of our follow-up, these procedures 
were being followed and that Assistive Devices Program staff were working to update all 
policy manuals, which would then be posted on the Program’s Web site. 


Pricing—Home Oxygen Program 


Recommendation 
To better ensure that the prices it is paying for home oxygen are reasonable, the Ministry should: 


* consider tendering for home oxygen on a test basis in larger urban centres; 


* closely monitor oxygen prices being paid by other provinces to ensure Ontario’s higher 
volumes are reflected in the comparative rates being paid; and 
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determine whether paying a single flat rate is more economical than negotiating 
different rates for liquid oxygen and concentrators. 


Current Status 


At the time of our follow-up, the Ministry indicated: 


It had discussed the feasibility of tendering for home oxygen in larger urban centres 
with home-oxygen vendors and other stakeholders. The Ministry informed us that, 
before negotiating the next agreement in fall 2003, the Ministry intends to prepare a 
business case and update the information on rates being paid by other provinces. 


Ministry staff intend to design a system for collecting statistics on the use of liquid 
oxygen and concentrators when preparing the strategy for negotiating the next 
agreement. 


Pricing—Assistive Devices Program 


Recommendation 


To help ensure that it is paying competitive prices for the devices funded under the Assistive 
Devices Program, the Ministry should: 


conduct a comprehensive review of the prices listed for all devices covered by the 
Assistive Devices Program; and 


obtain competitive bids from manufacturers or suppliers for devices that are similar in 
nature. 


Current Status 


According to the Ministry, at the time of our follow-up: 


A new pricing framework for the Assistive Devices Program had been approved. 


It had hired a consulting firm to complete a pricing review of high-risk and high-value 
products funded by the Program and to develop a mechanism for ongoing price 
updates. 


Ministry senior management was considering options for establishing a new equipment 
pool funded by the Ministry which would also provide comparative price quotations. 


Verification of Claims 


Recommendation 


To better ensure that individuals have actually received the devices and services paid for under its 
Assistive Devices and Home Oxygen programs, the Ministry should expand its verification letter 
process to cover all assistive device categories. The Ministry should also: 
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¢ track verification letters sent and replies received: 
° follow up on any discrepancies or non-replies on a timely basis; and 


° ensure that verification letters include sufficient detail to allow the Ministry to 
determine whether vendors are providing all the services and equipment required 
under their contracts. 


For claims submitted on computer disks, the Ministry should periodically inspect signed invoices 
maintained by the vendors. 


Current Status 
The Ministry advised us that, at the time of our follow-up: 


¢ The Finance and Audit Unit had established letter verification and tracking processes 
for all assistive device categories. 


¢ Ministry staff had started conducting regular site visits with selected vendors in 
September 2002. 


¢ Ministry staff complete monthly reports on suspected cases of fraud and recoveries to be 
pursued with vendors. Assistive Devices Program staff were working with the staff of the 
Ministry’s Fraud Program Branch to jointly identify policies and procedures that would 
assist in preventing fraud. 


Accountability of Transfer-payment Agencies 


Recommendation 
To help ensure that transfer-payment agencies funded by the Assistive Devices Program are 
providing services economically and efficiently, the Ministry should: 


°* ensure that it receives sufficient and appropriate financial information for assessing 


whether funds are being used for the purposes intended; 


¢ ensure that the distribution of funds is commensurate with the value of services 


provided; and 


° compare the costs to provide services among similar agencies. 


Current Status 


According to the Ministry, service agreements with transfer-payment agencies have been 
updated and the Ministry has requested that agencies provide additional reports with more 


detail on what services are being provided to clients. 
In addition, Assistive Devices Program staff had: 


* implemented a revised annual budget approval process for transfer-payment agencies in 
2001/02; 
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¢ implemented an annual training process relating to the budget submission process and 
new reporting requirements for agency staff and had completed two training sessions (all 
agencies are required to send one or two staff for training); and 


¢ developed statistical information packages to ensure consistent reporting by agencies 


and had trained agency staff in how to use them. 


Consulting Services 


Recommendation 


To better ensure that value for money is received when engaging consultants, the Ministry should 
ensure that: 


° consultants are engaged through a competitive process; and 


¢ long-term needs are addressed by hiring employees rather than engaging consultants at a 
rate of remuneration significantly exceeding the amounts paid to government employees 
performing similar duties. 


Current Status 


The Ministry advised us that a competitive process would be used for future arrangements 
with consultants. For instance, at the time of our follow-up, it had received approval to hire 
three system programmers, and competitions for these positions were in process. In the 
meantime, a competitive process had been used to hire, on an interim basis, three system 
programmers on a fee-for-service basis. 


CLAIMS APPROVAL, PROCESSING, AND PAYMENT 


Recommendation 


To help ensure that Assistive Devices and Home Oxygen program payments are made only for 
valid claims, the Ministry should implement procedures to run computer checks to identify 
payments made on behalf of individuals who are deceased. 


Current Status 

The Ministry advised us that, at the time of our audit, Assistive Devices Program staff were 
now receiving recoveries for deceased clients on a regular basis and that processes were now 
in place to ensure that files for deceased clients were designated inactive in a timely manner. 
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PERFORMANCE MEASUREMENT, MONITORING, 
AND EVALUATION 


Recommendation 
To better monitor and evaluate the performance of the Assistive Devices and Home Oxygen 
programs, the Ministry should: 


°* ensure its information systems provide accurate and timely reports on all key 
performance measures; and 


° reinstate the standing committees that provide technical advice for all major assistive 
device categories. 


Current Status 

According to the Ministry, it has developed performance measures and indicators, and 
monthly progress reports began to be generated and distributed internally in July 2002. 
The movement of all performance data to the new version of the information system was 
nearing completion. In addition, the Ministry has developed new tracking systems for 
financial forecasting, and internal reports are now available. 


As mentioned earlier, standing committees have been organized for the four medical device 
categories. At the time of our follow-up, the committees were meeting regularly to advise on 
operational policy changes. 


Complaints Process 


Recommendation 
To help identify any areas requiring improvements in the delivery of the Assistive Devices and 
Home Oxygen programs, the Ministry should ensure that: 


* complaints are investigated in a timely manner; and 


° the results of those investigations are provided to senior management. 


Current Status 


The Assistive Devices Program has implemented a new customer feedback/complaint 
resolution process that tracks issues raised by clients, vendors, individuals and organizations 
with authorizing authority, and others. This process also includes timelines for responses. 
Quarterly reports on complaints are provided to ministry senior management. 


The Ministry also indicated that, at the time of our follow-up, ministry staff had begun to 


design and implement a customer satisfaction survey. 
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OTHER MATTER 


Benefits from Other Government Sources 


Recommendation 


To help ensure that assistance under its Assistive Devices and Home Oxygen programs is not 
duplicated at taxpayers’ expense, the Ministry should again pursue co-operation with the 
Workplace Safety and Insurance Board and the Department of Veterans’ Affairs. 


Current Status 


According to the Ministry, at the time of our follow-up, its Fraud Programs Branch was 
pursuing co-operation with the Workplace Safety and Insurance Board and the 
Department of Veterans’ Affairs. 
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MINISTRY OF TRANSPORTATION 


4.11—Road User Safety Program 
(Follow-up to VFM Section 3.11, 2007 Annual Report) 





BACKGROUND 


The Ministry of Transportation’s goal is to foster improved road user safety and well-planned 
highway expansion and preservation to bolster provincial growth and development. The 
Ministry’s Safety and Regulation Division administers the Road User Safety Program by 
setting safety standards, enforcing compliance with those standards, testing and licensing 
drivers and vehicles, and educating road users about safe driving behaviour. 


‘To carry out its responsibilities, in 2001, the Ministry operated 48 driver examination 
centres and had contracted with 280 private issuing offices to provide driver and vehicle 
licence renewal and related services. During the 2000/01 fiscal year, the Ministry 
administered 611,000 road tests and processed over 18 million over-the-counter 
transactions. In the same year, the Program spent $101 million, and its licensing and 
registration activities generated approximately $894 million in revenue. 


During the course of our 2001 audit of the Road User Safety Program, the Ministry had 
not provided us with all the information and explanations needed to complete the audit. 
We subsequently received a commitment from both the new Minister and Deputy Minister 
of Transportation and saw evidence that corrective action would be taken to ensure that the 
access-to-information problems we had encountered would not reoccur in the future. 


Sufficient information and explanations were received in 2001 for us to conclude that in 
managing resources for its Road User Safety Program, the Ministry was deficient in ensuring 
due regard for economy and efficiency, it did not ensure compliance with programs 
designed to enhance road safety, and its procedures to measure and report on program 
effectiveness were not satisfactory. Our major concerns in 2001 included: 


¢ A significant number of applicants had to wait over six months to take a road test to try 
to obtain a driver’s licence. 


¢ Extreme variations in driver examiner pass rates were allowed to persist for over 10 years 
without corrective action being taken. 


° Road user safety was impaired because some drivers’ road tests had been shortened 
below the minimum standard time for properly evaluating necessary driving skills. 


¢ The Ministry hired 280 additional staff for driver testing in the 16 months ended 
January 2001 at a cost of $10.3 million and then opted for outsourcing driver testing to 
the private sector without a completed business case to justify this outsourcing. 
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¢ The Ministry planned to spend $101 million on computer systems work but, at the 
time of our audit, it did not provide us with assurance that the proposed system was 
supported by a properly completed business case. 


¢ Millions of dollars’ worth of consultants’ work was mismanaged as consultants were often 
selected without a competitive tendering process or engaged without a written contract 
in place. 


¢ Certain driver’s-licence suspensions for impaired driving were rescinded without follow- 
up because the Ministry did not receive prompt notification of the suspension from the 
police. 


¢ A backlog of 30,000 reports from medical practitioners had accumulated, some dating 
back as far as 1997, allowing drivers to operate vehicles even though they were reported 
to have conditions that could make it dangerous for them to do so. 


¢ The Ministry did not meet its annual reporting requirement on road user safety to the 
Legislature. At the time of our audit, the last report tabled was for 1997 and did not 
contain recommendations for the prevention of motor vehicle accidents as required by 


the Highway Traffic Act. 


¢ Customer service needed improving given that 49% of people responding to ministry 
comment cards were dissatisfied, mainly because of lengthy wait times, service not being 
prompt and efficient, and staff not being courteous and helpful. 


Accordingly, we made a number of recommendations to overcome these deficiencies and 
received commitments from the Ministry that it would take corrective action. 


CURRENT STATUS OF RECOMMENDATIONS 


The Ministry had taken or was in the process of taking corrective action on all of the 
recommendations made in our 2001 Annual Report. The current status of each of our 
recommendations is outlined below. 


DUE REGARD FOR ECONOMY AND EFFICIENCY 


Driver Examinations 


Recommendation 


To help ensure that the driver examination function of the Road User Safety Program efficiently 
meets its objective of passing only qualified individuals, the Ministry should: 


° periodically revise written tests; 


° achieve the six-week waiting time standard for road tests; 
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° implement procedures, including periodic training for examiners, to better achieve 
consistency in the application of driver test standards; and 


° adequately test and evaluate applicants in accordance with ministry driver test 
standards. 


Current Status 


Information provided by the Ministry indicates that waiting times to take a road test remain 
a chronic problem in some regions of the province. As of May 10, 2003 the average waiting 
period for all the regions was 8.3 weeks for the G1 driver’s test and 10.2 weeks for the G2 
driver's test. Some driver examination centres were experiencing wait times of up to 26 
weeks and 34 weeks for the G1 and G2 tests respectively. The Ministry indicated to us that 
it expects the six-week standard waiting time to be achieved when the new private-sector 
service provider takes over the driver examination centres in fall 2003. 


The Ministry also indicated that it had implemented ongoing reviews of written tests and 
has developed a bank of new questions that has been in use since January 2003. In addition, 
the application of driver-test standards and the testing of applicants is now monitored by 
management staff, and corrective action is taken if driver examiners do not consistently 


apply the standards. 


Alternative Service Delivery 


Recommendation 


To ensure that potential outsourcing alternatives for the driver examination function of the 
Road User Safety Program are analyzed in an objective and thorough manner, the Ministry 
should: 


° comply with the Management Board of Cabinet Framework on Alternative Service 
Delivery and ensure that a comprehensive business case is prepared prior to seeking 


approval for any option; and 


° ensure that documents submitted to Cabinet—upon which major decisions are to be 
based—are complete or clearly identify the limitations of the information. 


Current Status 


The Ministry informed us that in November 2001, it completed a business case for the 
alternative service delivery of the driver examination function and subsequently updated the 
financial information in the business case to March 31, 2002. The Ministry also indicated 
that, in December 2002, it reported the results of the bidding selection process to the 
Cabinet Committee on Privatization and SuperBuild. The report to the Committee 
included financial information from the Ministry's business case and a recommendation to 
proceed with awarding a contract to the successful bidder. 
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The successful bidder signed a 10-year licence agreement on February 24, 2003, and 
agreed to pay the Ministry $114 million, as well as $200,000 per month for compliance 
and audit monitoring. The agreement entitles the bidder to the driver examination fees over 
the 10-year period. The bidder paid the Ministry a deposit of $7.98 million and will pay 
the balance of $106.02 million when the driver examination function is transferred, which 


is expected to be in fall 2003. 


Information Technology 


As we reported in 2001, the Ministry had submitted a business case to Management Board 
of Cabinet for renewing the legacy system that supports the licensing and registration of 
drivers and vehicles. The total cost of the proposed renewal, which was to be completed in 
the 2003/04 fiscal year, was projected to be $101 million and was approved by the 
Management Board of Cabinet in December 2000. At the time of our audit, the Ministry 
did not provide us with the necessary supporting information for this project. Therefore, at 
that time, we could not provide assurance that the proposed legacy system renewal was 
based on a properly completed business case and strategic plan. 


Current Status 


The Ministry informed us that it has since prepared a new business case, which includes a 
detailed migration plan and the recommended design framework, which was 
communicated to Management Board of Cabinet through its 2002/03 and 2003/04 
Business Planning and Allocations process. Due to changes and refinements to some 
projects from the preliminary plans in the original business case, the completion date for the 
system renewal project has been revised to December 2005 and the cost revised to 

$112 million. 


Consulting Services 


Recommendation 
To ensure that consultants are engaged in a fair and competitive manner and that value for 


money is being received, the Ministry should: 


¢ properly justify the need for consultants’ services and document estimated costs before 
consultants are hired; 


¢ define the scope of the work and ensure contracts are signed before consultants begin 
their assignments; 


¢ select consultants based on an evaluation of the experience, qualification, and price of 
more than one supplier as required by Management Board of Cabinet directives, even 
if suppliers are selected from the vendor-of-record list prepared by the Management 
Board Secretariat; 


¢ not allow suppliers to gain a monopoly for any particular kind of work; 
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¢ help ensure fair competition by not awarding the same consultant successive agreements 
that cumulatively exceed the competition threshold of $25,000; 


¢ enforce the ceiling of contracts when the terms and conditions of an agreement remain 


unchanged; and 


¢ formally evaluate consultants when their assignments have been completed. 


Current Status 


The Ministry informed us that it has introduced a new monitoring system that will control 
the acquisition of consulting services from start to finish. As well, mandatory staff training 
was implemented, as was an acquisition/evaluation process checklist to ensure compliance 
with government procurement procedures. 


Private Issuers 


Recommendation 


To ensure that private issuers of the Road User Safety Program are properly managed, the 
Ministry should: 


° review the current agreements with issuers to incorporate the roles and responsibilities 
of both parties and include a non-performance clause, as well as make the Performance 
Management Program mandatory for all issuers; 


° impose penalties on issuers that, contrary to ministry procedures, consistently process 
transactions to increase CommMISSIONS; 


° hold issuers financially responsible for lost stock when the proper safeguarding 
procedures have not been followed; 


° hire new issuers on a timely basis to minimize disruptions to customer service; and 


° ensure that resources to train new issuers and monitor existing ones are adequate and 
consistent throughout the province. 


Current Status 


As part of its Private Issuer Project, the Ministry reviewed its current agreement with private 
issuing offices and revised it for new issuers. This agreement incorporates penalties for non- 
performance, the reimbursement of overpaid commissions, and a strengthening of the 
termination clause. However, the majority of issuers still operate under the old agreements, 
and the Ministry does not have a date for implementing the new requirements for this 
group. The Ministry informed us that the date would be determined during the second 


phase of the Private Issuer Project. 


With respect to hiring issuers on a timely basis, we reported in 2001 that it took up to six 
months to complete the selection process. The Ministry informed us that in 2002, seven 
new private issuers were selected through an enhanced procurement process. On average, it 
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took four months from the date the request for proposal was issued to the date the top 
bidder accepted the opportunity in writing. The Ministry ensured that there was no loss of 
service continuity in the areas affected. 


In November 2001, the Ministry issued stock and cash management procedures and 
guidelines that outline the issuers’ responsibilities for stock, cash, staff, and security. In May 
2003, the Ministry was still reviewing the security plans provided by issuers to ensure that 
the plans comply with the new policy. The Ministry is also developing a policy to hold issuers 
financially accountable and to charge them for the cost of investigating missing stock if the 
issuer was determined to be at fault. 


The Ministry informed us that it was now providing ongoing instruction to its Issuing 
Office Administrators, who monitor private issuing offices, to help ensure that all 
Administrators properly train all new private issuers. 


Revenue Collection and Control 


Recommendation 


To help ensure that the Ministry receives all the funds it is entitled to from driver licensing and 
vehicle registration fees, it should: 


° fully reconcile the transactions carried out and deposits made by private issuing offices 
on a timely basis and follow up on any discrepancies; 


° reinforce ministry cheque acceptance policies with private issuers and enforce measures 


to hold them accountable for any breach of policy; 


¢ fully examine current collection procedures for NSF (not sufficient funds) cheques and 
consider policy and/or legislative changes to improve them; and 


¢ transfer funds to be remitted to municipalities from the collection of parking fines only 
after cheques from vehicle owners have cleared the bank. 


Current Status 


The Ministry informed us that it has made a number of changes to ensure that it receives all 
funds it is entitled to from driver licensing and vehicle registration fees. These included: 
using an automated reconciliation program to reconcile transactions to deposits, establishing 
a more rigorous collection procedure for NSF (not sufficient funds) cheques, sending 
accounts older than two years to a collection agency, and pursuing policy options with the 
ministries of the Attorney General and Municipal Affairs and Housing for more timely 
remitting of outstanding parking fines to municipalities. 


The Ministry also informed us that, with the passage of the Government Efficiency Act, 
2002, which amended related parts of the Highway Traffic Act, it was developing a process 
for cancelling driver’s licences and denying vehicle plate renewals when individuals have 
outstanding debts with the Ministry. In this regard, the Ministry was reviewing potential 
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system changes to link its vehicle registration system with the driver licensing system to 
facilitate collection efforts. 


COMPLIANCE WITH POLICIES AND PROCEDURES 


Driver Performance Monitoring and Intervention 


Recommendation 


To help reduce the risk of motor vehicle accidents and improve driver performance, the Ministry 
should: 


° ensure that drunk-driver suspensions are only rescinded when such action is 


appropriately justified; 


° review and take action on reports of drivers who do not meet the minimum medical 
fitness standards on a more timely basis; 


° ensure that appropriate intervention for drivers who have accumulated excessive 
demerit points is undertaken on a timely basis; 


° develop standard training programs for driver improvement counsellors; and 


¢ ensure that counsellors have full information for making informed recommendations on 
potential licence suspensions and other remedial actions. 


Current Status 


The Ministry has made process and automated systems improvements to the Administrative 
Driver's Licence Suspension program so that drunk-driver suspensions are not rescinded as a 
result of notices not being received by the Ministry within the prescribed time. The 
Ministry informed us that for 2002, none of the suspensions under this program had been 
rescinded as a result of notices not being timely. 


With respect to medical fitness reports, the Ministry’s standard timeframe for processing 
these forms in order to suspend a driver’s licence is three to eight weeks. According to the 
Ministry, the backlog we noted in 2001 (the Ministry defines the backlog as cases older 
than six months) has been eliminated so that only current cases are now being processed. In 
addition, the Ministry has been working with the Ontario Medical Association to clarify the 
medical conditions that need to be reported and introduced a bill (Road Safety Act, 2003) 
in the Legislative Assembly in May 2003 to formalize these medical conditions and the 
reporting guidelines. 


As for the demerit-point system and related licence suspensions by driver improvement 
counsellors, the Ministry informed us that it hired additional staff-to ensure that backlogs do 
not occur. In addition, all counsellors have received additional training with respect to 
suspension standards and are now provided with full information on a driver's record, 
including information on criminal convictions. 
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Disabled Person’s Parking Permits 


Recommendation 
To ensure that only persons who are entitled to disabled person's parking permits receive them, the 
Ministry should: 


° verify that the person attesting to the applicant’s medical condition 1s a member of the 
medical profession; 


° consider requiring certification that the applicant's disability does not impair the 
person’s ability to safely operate a vehicle; and 


° consider verifying that the disabled parking permit holder is still entitled to a permit 
when the application for renewal is received. 


Current Status 

Effective March 1, 2002, the Ministry implemented a new renewal and medical verification 
process. Permit holders are now required to have a medical practitioner recertify that the 
medical condition that makes them eligible for a disabled person’s parking permit still exists 
at each renewal date. Also, the Ministry has outlined new offences and increased penalties 
for the misuse of permits in the Ontarians with Disabilities Act, 2001. 


MEASURING AND REPORTING ON PROGRAM 
EFFECTIVENESS 


Road Safety 


Recommendation 


To improve the effectiveness of the Road User Safety Program in making Ontario roads safer, the 
Ministry should: 


¢ include the required recommendations for the prevention of motor vehicle accidents in 
its Ontario Road Safety Annual Report and ensure the report is prepared on a timely 
basis; 


¢ review the driver-training time reductions within the Graduated Licensing System, 
update its Driver’s Handbook, assess the merits of public education programs for new 
drivers, and implement any necessary changes to help reduce the collision risks of novice 
drivers; and 


¢ carry out the planned study of its program for senior drivers to assess its effectiveness in 
identifying potentially unsafe drivers. 
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Current Status 


The Ministry’s latest Ontario Road Safety Annual Report was for 2001, and it is planning to 
issue the 2002 annual report by the end of 2003. The Ministry informed us that, 
subsequent to our audit, all of its annual safety reports contain recommendations for the 
prevention of motor vehicle accidents and indicate planned education and intervention 
programs for improving road user safety. 


To ensure that drivers are properly trained and tested to operate a motor vehicle, the 
Ministry updated its Drivers Handbook to reflect legislative changes to the rules of the road. 
The next planned update was scheduled for November 2003. In addition, the Ministry 
introduced a pilot public education program in April 2001 to reduce collision risk for 
novice drivers. By fall 2003, the Ministry intended to continue this program with 
educational videos and school and community programs. 


With respect to novice drivers, the Ministry informed us that an overall evaluation of the 
Graduated Licensing System, including the reduction in waiting time to take the next driver 
test that is granted to those who take an approved driver education course, was being 
completed by the Traffic Injury Research Foundation in partnership with the Insurance 
Bureau of Canada. A final report was expected in summer 2003. The Ministry also 
completed a study of its program for senior drivers and determined that the fatal collision 
rate for seniors had decreased since it had implemented biennial testing. 


Customer Service 


Recommendation 


To improve the customer service provided under its Road User Safety Program, the Ministry 
should: 


° analyze customer comments for common concerns; 
° establish a repository of best practices used by offices to address such concerns; and 


¢ establish internal performance benchmarks with respect to waiting times for over-the- 
counter services and periodically report on performance by office and region against 


established benchmarks. 


Current Status 


According to the Ministry, it completed an analysis of customer comments and identified 
best practices used by offices to deal with the concerns. This information, along with other 
research carried out by the Ministry, was being put into a repository, which is shared with 
regional staff to improve customer service. With respect to developing internal performance 
benchmarks for waiting times for over-the-counter services, the Ministry informed us that 
initiatives to improve customer service will be included in the second phase of the Private 
Issuers Project and are reflected in the contract for the alternative service delivery of the 
driver examination function. The contract stipulates that a customer’s wait time for service is 
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not to exceed twenty minutes. The Ministry intends to monitor this requirement and to take 


corrective action as necessary. 


Effectiveness Measures 


Recommendation 


To measure the effectiveness of the Road User Safety Program and its initiatives to achieve safer 
roads, the Ministry should: 


¢ develop goals or outcomes over which the Program has a reasonable degree of influence; 


and 


° develop performance measures that demonstrate how program initiatives contribute to 
the Program's goals or outcomes. 


Current Status 
In evaluating the Road User Safety Program’s performance, the Ministry will continue to 


measure its rate of fatalities per 10,000 licensed drivers and the percentage of customers 
satisfied with the services provided by the Private Issuing Network (as reported in the 
Ministry’s business plans). As well, the Ministry is now measuring and reporting on the 
percentage of vehicle occupants wearing seat belts. The Ministry acknowledges that the 
outcomes of these performance measures cannot be attributed exclusively to the Program 
but have detailed their contribution to achieving its goals in its business plans. The Ministry 
informed us that it is still evaluating current performance measures and best practices from 
other jurisdictions to help in developing additional program-level performance measures. 
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Chapter Five 


CHAPTER FIVE 


Public Accounts of the 
Province 





INTRODUCTION 


The Public Accounts for each fiscal year (the fiscal year ends March 31) are prepared under 
the direction of the Minister of Finance, as required by the Ministry of Treasury and 
Economics Act. The Act requires the Public Accounts to be delivered to the Lieutenant 
Governor in Council for presentation to the Legislative Assembly not later than the tenth 
day of the first session held in the following calendar year. This year, as a result of the election 
on October 2, 2003, the Public Accounts were not finalized until late in the fall. 


The financial statements of the province, which form part of the Public Accounts, are the 
responsibility of the Government of Ontario. This responsibility encompasses ensuring that 
the information in the statements, including the many amounts based on estimates and 
judgment, is presented fairly. The government is also responsible for ensuring that a system 
of control with supporting procedures is in place to provide assurance that transactions are 
authorized, assets are safeguarded, and proper records are maintained. 


Our Office audits the financial statements of the province. The objective of this audit is to 
obtain reasonable assurance that the financial statements are free of material misstatement— 
that is, that they are free of significant errors or omissions. The financial statements, along 
with the Auditor's Report on them, are included in the province’s annual report and 
provided in a separate volume of the Public Accounts. 


The province’s annual report contains, in addition to the financial statements, summaries 
and analyses of the province's financial condition and fiscal results. Providing such 
information enhances the fiscal accountability of the government to both the Legislative 
Assembly and the public. 


The Public Accounts also include three supplementary volumes: 


* Volume | contains the Consolidated Revenue Fund schedules and ministry statements. 
These schedules and statements reflect the financial activities of the government's 
ministries on a modified cash basis of accounting. 


* Volume 2 contains the financial statements of the significant provincial Crown 
corporations, boards, and commissions that are part of the government’s reporting 
enuty, as well as other miscellaneous financial statements. 
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¢ Volume 3 contains further details of public expenditures as well as the Ontario Public 
Service senior salary disclosure. 


Our Office reviews the information in the annual report and the three supplementary 
volumes for consistency with the information presented in the financial statements. 


Legislative changes to the Ministry of Treasury and Economics Act have been made with 
respect to the Public Accounts reporting process. Effective for the fiscal year commencing 
April 1, 2003, the province’s annual report, except in extraordinary circumstances, is to be 
delivered to the Lieutenant Governor in Council on or before the 180th day after the end 
of the fiscal year. The Lieutenant Governor in Council must then either lay the Public 
Accounts before the Assembly or, if the Assembly is not in session, make the Public Accounts 
public and, once the Assembly resumes sitting, lay them before the Assembly on or before 
the tenth day of the next session. Normally, the three supplementary volumes must be 
submitted to the Lieutenant Governor in Council before the 240th day after the end of the 
fiscal year. The Lieutenant Governor in Council must then lay the information before the 
Assembly or, if it is not in session, make the information public and then, when the 
Assembly resumes sitting, lay it before the Assembly on or before the tenth day of that 


session. 


THE PROVINCE’S 2002/03 FINANCIAL 
STATEMENTS 


The Audit Act requires that the Provincial Auditor report annually on the results of the 
Auditor’s examination of the province’s financial statements. This year, as a result of the 
Provincial Auditor’s retirement, it was my responsibility, in my capacity as the Assistant 
Provincial Auditor, to express an audit opinion on the financial statements. I am pleased to 
report that my Auditor’s Report to the Legislative Assembly on the financial statements for 
the year ended March 31, 2003 is clear of any qualifications or reservations and reads as 
follows: 


To the Legislative Assembly of the Province of Ontario 


| have audited the consolidated statement of financial position of the Province of Ontario 
as at March 31, 2003 and the consolidated statements of operations, change in net debt, 
and cash flow for the year then ended. These financial statements are the responsibility of 
the Government of Ontario. My responsibility is to express an opinion on these financial 
statements based on my audit. 


| conducted my audit in accordance with Canadian generally accepted auditing 
standards. Those standards require that | plan and perform an audit to obtain reasonable 
assurance whether the financial statements are free of material misstatement. An audit 
includes examining, on a test basis, evidence supporting the amounts and disclosures in 
the financial statements. The audit also includes assessing the accounting principles 
used and significant estimates made by the Government, as well as evaluating the overall 
financial statement presentation. 
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In my opinion, these financial statements present fairly, in all material respects, the 
financial position of the Province as at March 31, 2003 and the results of its operations, 
the changes in its net debt, and its cash flows for the year then ended in accordance with 
accounting principles recommended for governments by the Canadian Institute of 
Chartered Accountants. 


[signed] 
Toronto, Ontario Jim McCarter, CA 
September 19, 2003 Assistant Provincial Auditor 


ACCOUNTING FOR CAPITAL ASSETS 


The Ontario government prepares its financial statements in accordance with the 
accounting principles recommended for governments by the Public Sector Accounting 
Board (PSAB) of the Canadian Institute of Chartered Accountants. In January 2003, PSAB 
revised a 1997 standard setting out rules for the recognition, measurement, amortization, 
and presentation of capital assets in a government's financial statements. Until recent years, 
most governments, including that of Ontario, have charged to operations 100% of the cost 
of capital assets in the year the capital assets were acquired or constructed. The revised 
standard recommends that capital assets be recorded as assets in the financial statements at 
cost and be amortized to expense over their estimated useful lives (this is the basis of 
accounting for capital assets followed in the private sector). Related revisions recommend 
that the government report both its net debt and its accumulated deficit (net debt minus 
non-financial assets) on its statement of financial position and that the accumulated deficit 
accordingly be reduced by the unamortized value of government capital assets. 


In its spring 2001 Budget, the Ontario government announced its intention to adopt the 
PSAB recommendations relating to tangible capital assets. The government began phasing 
in this approach for the 2002/03 fiscal year under the direction of the Provincial Controller, 
concentrating its efforts on capitalizing its land holdings, buildings, and transportation 
infrastructure. Our Office supports the government’s phased-in approach as a reasonable 
method for implementing PSAB’s capital asset standards. The result of the exercise this year 
has led to the recognition, for the first time in the government’s accounts, of net capital 
investments of over $13 billion. This accounts for an estimated 90% or more of the 
government's total tangible capital assets. 


Ontario’s remaining tangible capital assets, such as computer systems, vehicles and 
equipment, and other smaller-value items, will be identified, valued, and included in the 
government's financial statements in subsequent years. PSAB has specifically excluded from 
its current standard the need to address and value such difficult-to-capitalize assets as works 
of art, historical treasures, natural resources, and Crown lands not acquired via government 
purchase. 
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LAND HOLDINGS AND BUILDINGS 


The values for most of the government’s land holdings and buildings were arrived at by the 
Ontario Realty Corporation (ORC)—the government agency responsible for managing the 
government's real estate assets. ORC determined the values by launching a Book Value 
Project (BVP) in January 2002. The objective of the BVP was to estimate, using the best 
evidence available, both the original cost of government land and buildings and the dates of 
their acquisition (or, where applicable, the start and end dates of their construction). Four 
primary approaches were used to obtain the estimates of original cost: 


¢ ORC data were compared with the values for land and buildings that the Municipal 


Property Assessment Corporation provided for municipal taxation purposes. 
¢ ORC historical asset records were examined. 


¢ Cost estimates for buildings were developed internally on the basis of staff knowledge of 
the buildings and a review and analysis of building characteristics such as size, location, 
age, and condition. 


¢ Third-party appraisals were used. 


Once the original cost of the land and buildings had been estimated, a valuation software 
program called the Book Value Calculator (BVC)—originally developed by the federal 
government to provide valuations of federally owned capital assets—was used to estimate 
current unamortized values for the buildings. The BVC did not need to calculate 
amortization for land, since land has an unlimited useful life and is therefore not subject to 
the deterioration or depreciation over time that amortization is intended to capture. For 
each building, the BVC calculated amortization by combining information on the 
government's historical rate of reinvestment in its buildings, the building's age, and its 
estimated useful life. With the BVC, ORC was thus able to arrive at opening book values for 
Ontario’s buildings as at April 1, 2002 on both a gross basis (before accumulated 


amortization) and a net basis (after accumulated amortization). 


TRANSPORTATION INFRASTRUCTURE 


In January 2002, the Ministry of Finance (MOF) and the Ministry of Transportation 
(MTO) contracted with an outside consulting firm to help provide—for the province's 
highways, other roads, bridges, and associated assets such as road signage and lights—an 
estimate of the opening book values and amortization expense for the fiscal year ending 
March 31, 2003. Consistent with the phased-in approach discussed earlier in this section, 
MTO equipment was specifically excluded from the scope of this exercise. Rather than 
inventory individual items in the transportation infrastructure, as was done for land and 
buildings, MTO allocated its capital expenditures over the last 60 years into nine asset 
categories. Accumulated amortization to date was then estimated for each of these asset 
categories based on the estimated useful life attributable to that category. 
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RELIABILITY OF RESULTS 


We acknowledge that, whenever estimates are used to determine financial statement 
amounts, it is possible that different estimation approaches could yield different results. 
However, our audit work on the estimation and valuation process used by the government 
and on the calculations of gross and net book values indicated that the values arrived at are 
reasonable. Over the years the accuracy of the province’s capital asset information will 
steadily improve as all capital assets are recorded, the estimated opening book values are 
amortized, assets are gradually replaced, and the actual dollars spent on these assets are 
capitalized. 


STRANDED DEBT OF THE 
ELECTRICITY SECTOR 


Since our 1998 Annual Report, we have raised issues relating to the restructuring of the 
electricity sector. Of particular concern to us has been the risk to Ontario’s taxpayers with 
respect to the stranded debt of approximately $20 billion that resulted from the 
restructuring of the sector. This debt arose because, in a new competitive environment, the 
net asset values in Ontario Hydro’s successor companies as at April 1, 1999 were 

$19.4 billion lower than the net assets previously recorded in Hydro’s accounts. An agency 
of the province—the Ontario Electricity Financial Corporation (OEFC)—is responsible for 
managing the stranded debt. 


Although this stranded debt had been recorded as a liability in the government's financial 
statements, it was separated from other liabilities on the province’s statement of financial 
position. Similarly, while the electricity sector’s net operating results were included in the 
province's annual surplus or deficit, they were disclosed separately from the results of other 
government operations on the province's statement of operations. This separation existed 
because the government plans to recover this debt from electricity ratepayers rather than 
taxpayers. 


Developments in the current fiscal year indicated that there continued to be an increasing 
risk that part or all of the stranded debt would not be recovered from the electricity 
ratepayers and therefore would become a liability of the taxpayers of the province. For 
example, during the 2002/03 fiscal year, the stranded debt liability increased by another 
$98 million. In fact, the stranded debt has increased almost every year since April 1, 1999, 
when the electricity sector was restructured, as reflected in the following table: 


eee er 
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Fiscal Year Stranded Debt 
End Date ($ billion) 


April 1, 1999 19.433 


March 31, 2003 


Prepared by Office of the 
Provincial Auditor 





When the stranded debt was assumed by OEFC, the government established a long-term 
plan to retire the debt solely from dedicated revenue streams derived from the electricity 
sector. The government's long-term plan is updated annually to reflect current information 
and assumptions. As with any long-term plan, there is a high degree of uncertainty as to 
whether forecasted results will be achieved. The following examples illustrate the 
uncertainties inherent in electricity sector forecasting: 


¢ The actual pre-tax profits of Ontario Power Generation (OPG) and Hydro One, which 
go towards reducing the stranded debt, were $870 million lower in the four fiscal years 
from 1999/2000 to 2002/03 than the amount forecasted in the initial government 
long-term plan. This decrease was primarily due to lower-than-expected OPG profits 
resulting from cost overruns from ongoing repairs on the Pickering nuclear facility. The 
Pickering cost overruns have also contributed to OPG having to defer, for another two 


years, principal payments due to OEFC amounting to $700 million. 


¢ Effective May 1, 2002, the government fixed the electricity price paid by low-volume 
and designated consumers at 4.3 cents per kilowatt hour until at least 2006. For the 
2002/03 fiscal year, this resulted in $665 million in additional costs. A price cap of this 


nature was not envisioned in the initial long-term plan. 


This year, in part because of the increased concerns that the taxpayers of the province rather 
than electricity ratepayers will ultimately be responsible for repaying the stranded debt, the 
government has changed its treatment of the electricity sector for financial statement 
purposes. While the government still intends for the stranded debt to be recovered from 
electricity ratepayers, the financial statements for the 2002/03 fiscal year provide better 
disclosure of the activities of the government, in accordance with PSAB recommendations, 
by combining the electricity sector liabilities, including the stranded debt, with other 
government liabilities. We support this change. 
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CANADA HEALTH AND SOCIAL 
TRANSFER SUPPLEMENT 


In its 2003 Budget, the federal government announced a $2.5 billion supplement to the 
Canada Health and Social Transfer (CHST) to assist provinces and territories in addressing 
health care needs. Ontario’s share of this supplement was $967 million. The transfer was 
intended to provide funding over the three-year period from the 2003/04 fiscal year 
through the 2005/06 fiscal year. As with previous supplements, these federal funds were 
deposited into a trust account after the March 31, 2003 federal fiscal year end. As part of 
the trust agreement, a schedule was provided to provinces suggesting how the funds should 
be drawn down over the three-year period covered by the transfer. However, as occurred 
with past supplements, provinces and territories were free to draw down their portion of the 
$2.5 billion as they saw fit to meet their specific needs. The federal funds were deposited 
into the trust in June of 2003, and the Ontario government, as it had done with past 
supplements, withdrew its $967-million allocation in full shortly thereafter. 


Historically, the Ontario government has recognized CHST transfers as provincial revenue 
over the periods identified in the federal government’s drawdown schedule. This 
accounting policy is outlined in the notes to the government’s financial statements and has 
been accepted by our Office. Although the government’s 2003 Budget reflected the entire 
amount of the $967 million transfer as revenue in the 2002/03 fiscal year, the audited 
financial statements for 2002/03 appropriately recognized no revenue relating to this 
CHST transfer, since the federal government's schedule did not identify the 2002/03 
period for drawdown. Based on the government's stated accounting policy, we anticipate 
the government will recognize $386 million of the total $967 million transfer as revenue in 
the 2003/04 fiscal year and the remaining $581 million as revenue over the following two 
fiscal years. 


THE GOVERNMENT REPORTING 
ENTITY 


The “government reporting entity” refers to, collectively, all of the organizations whose 
activities are included in the government’s financial statements. One of the most critical 
aspects of reporting on a government’s financial affairs is deciding which organizations— 
from among, for example, ministries, agencies, Crown-controlled corporations, boards, 
commissions, and organizations recetving transfer payments—should be included in the 
reporting entity. Inclusion in the reporting entity essentially means that an organization’s 
operating results and its assets and liabilities are consolidated with or otherwise incorporated 
into the government's financial statements so that they form part of both the government’s 
annual deficit or surplus and its accumulated deficit or surplus. 


‘To date, the government has used a longstanding PSAB standard in assessing which 
organizations to include in its reporting entity. This standard recommends that an 
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organization be included in the government’s financial statements if: (1) it is accountable for 
the administration of its financial affairs and resources either to a minister of the government 
or directly to the Legislature and (2) it is owned or controlled by the government. In 
accordance with that standard, Ontario’s financial statements for the fiscal year ended 
March 31, 2003 include, in addition to the activities of all government ministries, those of 
27 of its most significant organizations, such as Ontario Power Generation Inc., Hydro One 
Inc., the Ontario Electricity Financial Corporation, the Ontario Lottery and Gaming 
Corporation, the Liquor Control Board of Ontario, GO Transit, and the Ontario Housing 
Corporation. The activities of less significant government organizations are accounted for 
through the accounts of the ministries responsible for them. 


It is important to note that, under the current standard, the reporting entity does not 
include the activities of a number of significant public-sector or quasi-public-sector 
institutions that operate outside of the government ministry and agency structure. Most of 
these institutions, however, are primarily funded by the government and have considerable 
assets, liabilities (including long-term debt), revenues, and expenditures. Historically, the 
majority of these institutions form what has been called the “SUCH” sector, where SUCH 
stands for school boards, universities, colleges, and hospitals (including long-term-care 
facilities). Including such agencies in the government's financial statements would have a 
significant impact on the province’s reported financial position and its annual operating 
results. 


Ontario also does not include in its reporting entity a major special-purpose agency—the 
Ontario Innovation Trust (Trust). The Trust has received large sums of money from the 
government under a mandate to eventually pass on these funds via grants or transfers, often 
years later, to the actual intended recipients to support specific government programs. 
Although technically the Trust does not meet the current PSAB standard for inclusion in 
the reporting entity, our Office has raised accountability concerns with respect to the trust 
in each of the last four annual reports of the Provincial Auditor, including section 3.07 of 
Chapter Three of this year’s report. 


In August 2003, PSAB revised its reporting-entity standard. The new standard 
recommends that government financial statements include the activities of all organizations 
that are controlled by the government. The standard offers extensive guidance in assessing, 
using “control indicators,” the degree to which control exists in any particular situation. The 
recommended implementation date for the standard is April 1, 2005. 


Since raising reporting-entity concerns in last year’s Annual Report, the staff at our Office 
have examined various sources of evidence—such as legislation, regulations, reporting 
arrangements, and ministry accountability documents pertaining to SUCH-sector 
entities—to assess whether the government will need to consider expanding its reporting 
entity under the new standard. It should be pointed out that applying PSAB’s indicators of 
control is not an exact science and requires the exercise of professional judgment in 
determining the applicability of the control indicators. It has also often been necessary to 
assess the substance of the relationship between government and the organization in 
addition to its legal form. The following chart summarizes our assessment. 
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Status of SUCH-sector Organizations 
With Respect to PSAB Control Indicators 


Govt. can unilaterally 
appoint/remove majority of 
members of the organization’s 
governing body. 


Govt. has ongoing access to 
the organization’s assets, 
ability to direct ongoing use of 
assets, or has ongoing 
responsibility for losses. 


Govt. holds majority of voting 
shares or “golden shares” 
giving power to govern 
organization’s financial and 
operating policies. 


Govt. has unilateral power to 
dissolve the organization and 
thereby access its assets and 
become responsible for 
organization’s obligations. 


Other Indicators 


Govt. has power to provide 
significant input into 
appointment of members to the 
organization’s governing body. 
Govt. has power to 
appoint/remove CEO or other 
key personnel. 


Govt. can establish/amend the 
organization’s 
mission/mandate. 

Govt. has power to approve 
the organization’s business 
plans or budgets and call for 
amendments. 


Govt. has power to establish 
borrowing/investment limits or 
restrict the organization’s 
investments. 


Govt. can restrict the revenue- 
generating capacity of the 
organization including its 
sources of revenue. 


Govt. can establish/amend the 
organization’s managing 
policies including those relating 
to accounting, personnel, 
compensation, collective 
bargaining, and deployment of 
resources. 


Y = control indicator met 





a 
oe 
pea 
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Based on our assessment, the two strongest candidates for future consolidation into the 
government reporting entity are Ontario’s colleges and school boards. Colleges in particular 
likely warrant inclusion, insofar as they meet the most control indicators, both primary and 
other. Although school boards meet only one of the primary indicators, they meet most of 
the other indicators, and the government has significantly increased its control over the 
delivery of Ontario’s elementary and secondary school programs in recent years. For 
example, the provincial government, rather than local boards, now sets the local education 
tax rate. Moreover, such education taxes, raised from the local property tax base, are no 
longer provided directly to the local school boards for use in local schools. Rather, the 
Ontario government now redistributes such taxes throughout the province on a centrally 
determined basis. The government has further tightened control over school-board 
operations by establishing legislation and policies in such key areas as allowable average class 
sizes, hours of instruction, province-wide curricula, and student testing. The government 
demonstrated its control over the boards last year by directly taking over three of the 
province’s largest boards. In addition, now that the province has adopted new PSAB 
accounting rules for tangible capital assets (see the previous section on “Accounting for 
Capital Assets”), including the financial statements of school boards in the financial 
statements of the province would allow the boards’ tangible capital assets to be included as 
well as their net debt, which was incurred to construct these capital assets. 


In conclusion, we recommend that the government complete its own assessment of colleges 
and school boards as soon as possible to determine if colleges and school boards should be 
consolidated into Ontario’s financial statements in time for the April 1, 2005 PSAB- 
standard implementation date. We would also encourage the government to complete an 
assessment of Ontario’s hospital sector and its long-term-care facilities to determine whether 
these organizations are sufficiently controlled by the government to warrant inclusion. 
Lastly, although we continue to have accountability concerns regarding the operations of 
the Ontario Innovation Trust, we continue to support the exclusion of both the Trust and 
Ontario’s universities from the government's reporting entity, since these organizations do 
not, in our opinion, meet PSAB’s control criteria. 


INTEGRATED FINANCIAL 
INFORMATION SYSTEM 


In the summer of 1995, the Minister of Finance established the Ontario Financial Review 
Commission (Commission) to review the government's financial practices. In its report 
issued November 1995, the Commission made 55 recommendations covering three main 
areas: planning, financial reporting, and Crown agencies. Among other things, the 
Commission noted that government ministries were using a number of different accounting 
and financial management systems, leading to financial information that was often 
inconsistent, delayed, or unnecessarily duplicated. It recommended that the government 
adopt one financial management and reporting system to replace the different, largely 
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incompatible systems that were being used throughout the government. This concept of an 
enterprise-wide financial system was also critical to the implementation of other 
recommendations made by the Commission relating to the government’s budget, its 
quarterly financial statements, and the future adoption of PSAB’s standards on capital assets. 
Addressing these areas required information that could not readily be produced by the 
government's existing financial systems. 


In late 1998, the Management Board of Cabinet approved the Ministry of Finance’s 
business case for an integrated financial system, and gave approval to the Ministry to 
procure the hardware, software, and expert services necessary to develop and implement the 
system. The resulting system, known as the Integrated Financial Information System (IFIS), 
is currently being implemented in phases, or releases, throughout the Ontario public 
service. IFIS was first released to two pilot ministries in November 2002. Since then, the 
government has released IFIS in waves, or at specified intervals, to several other ministries. 
Wave 1 and 2 implementations occurred during the winter and spring of 2003, 
respectively, resulting in seven additional ministries adopting the IFIS system. Future plans 
call for IFIS to be implemented in approximately 70% of Ontario’s ministries—responsible 
for just under one-half of the government's total expenditures—by March 31, 2004. The 
plan is for IFIS to be fully implemented across all Ontario ministries by the fall of 2004. 


NEW PSAB INITIATIVES 


PSAB serves the interests of the public and of the accounting profession by developing and 
promulgating standards designed to improve the financial and performance information 
reported by governments and other public sector entities. Improved information benefits 
decision-makers and other users. 


From Ontario’s perspective, some of the most significant issues PSAB is dealing with at the 
present time are the following: 


¢ In October 2002, PSAB approved a Statement of Principles that proposed revisions to 
the existing standard on how government transfers of funds to other organizations or 
levels of government should be accounted for. One of the more difficult issues to be 
resolved deals with multi-year funding—that is, funding that is provided to 
organizations or other levels of government in advance of the years the funds will be 
spent to provide services to the public. The accounting issue relates to whether 100% of 
the funds transferred should be recognized as expenses in the year of the transfer or be 
recognized as expenses when the services are actually provided to the public. 
Fundamental to this debate is whether financial statements should have an “asset-and- 
liability” focus or a “revenue-and-expense” focus: 


- Proponents of the asset-and-liability focus believe that transfers covering multiple 
periods should be charged to operations in one period—the period in which the 


transfer is authorized, eligibility criteria have been met, and the amount is estimable. 
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Pull recognition of the transfer is seen as appropriate because multi-year funding 
does not meet the definition of an asset, and recording a portion of it as such would 
distort a government’s statement of financial position. 


- Proponents of the revenue-and-expense focus believe that expensing multi-year 
funding in one year results in a distortion of the operating statement and therefore 
of the reported deficit or surplus of the transferring government over the periods 
funded. Their view is that multi-year funding should be charged to operations only 
when the funds are actually spent to provide services to the public, with the funds 


provided in advance of current-year needs being treated as prepaid assets. 


In 2002, PSAB also approved a Statement of Principles related to three financial- 
statement components: liabilities, contingent liabilities, and commitments. The 
Statement proposed changes to the definitions of each of these components. For 
example, the liability definition was broadened to include obligations that result from 
transactions and events beyond those related to agreements, contracts, and existing 
legislation. Issues addressed in the Statement included: whether legislation had to be in 
force prior to the accounting date before a liability could be recognized; the accrual of a 
contingent liability when the future confirming event is likely to happen; and limiting 
the disclosure of commitments to a government's contractual obligations. 


In July 2003, PSAB issued a draft guideline on funds and reserves that provides 
guidance on presenting information related to stabilization funds and financial reserves 
in government financial statements. 


In July 2003, PSAB issued a public exposure draft for a proposed Statement of 
Recommended Practices on Financial Statement Discussion and Analysis (FSD&A). 
The draft provides a general framework for the development of FSD&A information 
and provides guidance on the nature and extent of supplementary information that 
should be provided to financial statement users. This supplementary information would 
include narrative explanations and graphic illustrations of key events during the 
reporting period, along with explanations and illustrations of variances and trends. 


In August 2003, PSAB issued a Statement of Principles for its Generally Accepted 
Accounting Principles (GAAP). This Statement is the first step in the development of a 
proposed new GAAP section in the Public Sector Accounting Handbook (Handbook). 
The section would revise existing material in the Handbook relating to what other 
accounting guidance could be used when a particular accounting issue is not addressed 


within the Handbook itself. 


In April 2003, based primarily on input from the government community, PSAB 
approved four new projects, the following three of which have possible implications for 
Ontario’s financial reporting: 

- Financial Instruments—This project was begun to deal with the growth in the 


availability and use by governments of sophisticated financial instruments such as 
foreign currency and interest rate swap and option contracts. 
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- Performance Reporting—TVhis project is designed to develop a set of overarching 
principles that will guide the future development of performance reporting, 
including a framework for identifying specific performance indicators. 


-  Revenues—This project will define government revenue, consider the 
appropriateness for governments of the general principle of revenue recognition 
embodied in existing standards, and develop specific recognition criteria for 
government exchange and non-exchange transactions. 


OTHER MATTERS 


The Provincial Auditor is required under section 12 of the Audit Act to report on any 
Special Warrants and Treasury Board Orders issued during the year. In addition, under 
section 91 of the Legislative Assembly Act, the Provincial Auditor is required to report on any 
transfers of money between items within the same vote in the Estimates of the Office of the 


Legislative Assembly. 


LEGISLATIVE APPROVAL OF GOVERNMENT 
EXPENDITURES 


The government tables detailed Expenditure Estimates, outlining each ministry’s spending 
proposals on a program-by-program basis, shortly after presenting its Budget. The Standing 
Committee on Estimates reviews selected ministry Estimates and presents a report on them 
to the Legislature. The Estimates of those ministries that are not selected for review are 
deemed to be passed by the Committee and are reported as such to the Legislature. Orders 
for Concurrence for each of the Estimates reported on by the Committee are debated in 
the Legislature for a maximum of three hours and then voted on. 


Once the Orders for Concurrence are approved, the Legislature provides the government 
with legal spending authority by approving a Supply Act, which stipulates the amounts that 
can be spent according to the ministry programs as set out in the Estimates. Once the 
Supply Act is approved, the individual program expenditures are considered to be Voted 
Appropriations. The Supply Act, 2002 pertaining to the fiscal year ended March 31, 2003, 
received Royal Assent on December 13, 2002. 


‘Typically, prior to the passage of the Supply Act, the Legislature authorizes payments by 
means of motions for interim supply. For the 2002/03 fiscal year, the time periods covered 
by the motions for interim supply and the dates that the motions were agreed to by the 
Legislature were as follows: 


¢ November 1, 2001 to April 30, 2002—passed October 22, 2001; 


¢ May 1, 2002 to October 31, 2002—passed June 26, 2002 (see “Special Warrants”); 
and 


* November 1, 2002 to April 30, 2003—passed October 22, 2002. 
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SPECIAL WARRANTS 


If motions for interim supply cannot be approved because the Legislature is not in session, 
section 7 of the Treasury Board Act, 1991 allows the issue of Special Warrants authorizing 
the expenditure of money for which there is no appropriation by the Legislature. Special 
Warrants are authorized by Orders-in-Council approved by the Lieutenant Governor on 
the recommendation of the government. 


One Special Warrant was issued for the fiscal year ended March 31, 2003. The Special 
Warrant was approved by an Order-in-Council, dated April 24; 2002, totaling 
$19,630,031,900. The Special Warrant was required because the October 22, 2001 
motion for interim supply covered the period to April 30, 2002 only. Therefore, a Special 
Warrant was needed to authorize expenditures after that date until a new motion of interim 
supply could be passed (the new motion was passed on June 26, 2002). This Special 
Warrant authorized payments for both the general and necessary expenditures of the 
government and the general and necessary expenditures of the Office of the Chief Election 
Officer, the Provincial Auditor, the Legislative Assembly, and Ombudsman Ontario for a 


period of three consecutive months, commencing on May 1, 2002. 


The total expenditures approved by the Supply Act, 2002 excluded the amounts authorized 
by the Special Warrant. 


TREASURY BOARD ORDERS 


Section 8 of the Treasury Board Act, 1991 allows the Treasury Board to make an order 
authorizing payments to supplement the amount of any Voted Appropriation that is 
insufficient to carry out the purpose for which it was made, provided the amount of the 
increase is offset by a corresponding reduction of expenditures from other Voted 
Appropriations not fully spent in the fiscal year. The order may be made at any time before 
the first day of May following the end of the fiscal year in which the supplemented 


appropriation was made. 


The following chart is a summary of the total value of Treasury Board Orders issued for the 
past five fiscal years: 
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Treasury Board Orders for the 2002/03 fiscal year summarized by month of issue are as 
follows: 


640,808,400 





March 2003 
April 2003 








In accordance with a Standing Order of the Legislative Assembly, the preceding Treasury 
Board Orders are to be printed in The Ontario Gazette in the fall of 2003, together with 
explanatory information. A detailed listing of 2002/03 Treasury Board Orders, showing the 
amounts authorized and expended, is included as Exhibit Three of this report. 


TRANSFERS AUTHORIZED BY THE BOARD OF 
INTERNAL ECONOMY 


When the Board of Internal Economy authorizes the transfer of money from one item of 
the Estimates of the Office of the Assembly to another item within the same vote, section 91 
of the Legislative Assembly Act requires the Provincial Auditor to make special mention of the 
transfer(s) in the Annual Report. 
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With respect to the 2002/03 Estimates, the following transfers were made: 
¢ within Vote 201: 


From: Item6 Sergeant at Arms and Precinct Properties $ 461,300 
TO: ltem2 Office of the Clerk 330,800 
Item 7 Legislative Information Systems 126,400 
ltem 12 Lieutenant Governor's Suite 4,100 


¢ within Vote 202: 


From: Item3 Office of the Integrity Commissioner $ 63,400 
To: Item 1 Environmental Commissioner 9,100 
Item 2 Office of the Information and Privacy Commissioner 54,300 


UNCOLLECTIBLE ACCOUNTS 


Under section 5 of the Financial Administration Act, the Lieutenant Governor in Council, 
on the recommendation of the Minister of Finance, may authorize an Order-in-Council to 
delete from the accounts any amount due to the Crown which is deemed to be 
uncollectible. The losses deleted from the accounts during any fiscal year are to be reported 
in the Public Accounts. 


In the 2002/03 fiscal year, receivables of $84.8 million due to the Crown from individuals 
and non-government organizations were written off (in 2001/02 the comparable amount 
was $126.5 million). Volume 2 of the 2002/03 Public Accounts of Ontario provides a listing 
of these write-offs in total by ministry or Crown agency. 


Under the accounting policies followed in the audited financial statements of the province, 
a provision for doubtful accounts is recorded against the accounts receivable balances. 
Accordingly, most of the $84.8 million in write-offs had already been provided for in the 
audited financial statements. However, the actual deletion from the accounts required 


Order-in-Council approval. 

The major portion of the write-offs related to the following: 

¢ $35.9 million for uncollectible taxes relating to retail sales tax receivables; 

¢ $21.5 million for uncollectible taxes relating to corporation tax receivables; 


¢ $7.8 million for uncollectible assessments under the Ontario Disability Support 
Program; 


¢ $4.9 million for uncollectible assessments under the student support programs; and 


¢ $3.5 million for uncollectable taxes relating to employer health tax receivables. 
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CHAPTER SIX 


The Office of the 
Provincial Auditor 


MISSION STATEMENT 


Our mission is to report to the Legislative Assembly objective information and 


recommendations resulting from our independent audits of the government’s programs and its 
Crown agencies and corporations. In doing so, the Office assists the Assembly in holding the 
government and its administrators accountable for the quality of the administration’s 
stewardship of public funds and for the achievement of value for money in government 
operations. 





INDEPENDENCE 


The Provincial Auditor is appointed as an officer of the Legislative Assembly by the 
Lieutenant Governor in Council on the address of the Assembly. This is done after 
consultation with the Chair of the Standing Committee on Public Accounts (for more 
information on the Committee, see Chapter Seven). The Provincial Auditor and staff of the 
Office are independent of the government and its administration. This independence is a 
safeguard that enables the Office to fulfill its auditing and reporting responsibilities 
objectively and fairly. 


The Board of Internal Economy—an all-party legislative committee that is independent of 
the government's administrative process—reviews and approves our budget, which is 
subsequently laid before the Legislative Assembly. As required by the Audit Act, the Office’s 
expenditures relating to the 2002/03 fiscal year have been audited by a firm of chartered 
accountants and are presented at the end of this chapter. The audited statement of 
expenditure is submitted annually to the Board and subsequently tabled in the Legislative 
Assembly, 


AUDIT RESPONSIBILITIES 


We audit the financial statements of the province and the accounts of many agencies of the 
Crown, and we audit the administration of government programs and activities, as carried 
out by ministries and agencies of the Crown under government policies and legislation. Our 
responsibilities are set out in the Audit Act (see Exhibit Four). 
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The Office reports on its audits in an Annual Report to the Legislative Assembly. In 
addition, the Office may make a special report to the Assembly at any time on any matter 
that in the opinion of the Provincial Auditor should not be deferred until the Annual 
Report. We also assist and advise the Standing Committee on Public Accounts in its review 
of the Annual Report of the Provincial Auditor and of the Public Accounts of the province. 


It should be noted that our audit activities include the examination of the application of the 
government's administrative policies and the actual administration of the government's 
policy decisions as carried out by ministry or agency management. The government is held 
accountable for policy matters by the Legislative Assembly, which continually monitors and 
challenges government policies and programs through questions during legislative sessions 
and through reviews of legislation and expenditure estimates. 


We are entitled to have access to all relevant information and records necessary to the 
performance of our duties under the Audit Act. Out of respect for the principle of Cabinet 
privilege, the Office does not seek access to the deliberations of Cabinet. However, the 
Office can access all other information contained in Cabinet submissions that we deem 
necessary to fulfill our auditing and reporting responsibilities under the Audit Act. 


ACCOUNTS OF THE PROVINCE AND VALUE FOR 
MONEY 


The Provincial Auditor, under subsection 9(1) of the Audit Act, is required to audit the 
accounts and records of the receipt and disbursement of public money forming part of the 
Consolidated Revenue Fund whether held in trust or otherwise. To this end, the Office 
carries out an annual attest audit to enable the Provincial Auditor to express an opinion on 
the province’s financial statements. As well, the Office carries out cyclical value-for-money 
audits in accordance with subsection 12(2) of the Audit Act. 


AGENCIES OF THE CROWN AND CROWN- 
CONTROLLED CORPORATIONS 


The Provincial Auditor, under subsection 9(2) of the Audit Act, is required to audit those 
agencies of the Crown that are not audited by another auditor. Exhibit Two, Part (1), lists the 
agencies that were audited during the 2002/03 audit year. Public accounting firms are 
currently contracted by the Office to audit the financial statements of several of these 
agencies on its behalf. 


Exhibit Two, Part (I) and Exhibit Three list the agencies of the Crown and Crown- 

controlled corporations, respectively, that were audited by public accounting firms during 
the 2002/03 audit year. Subsection 9(2) of the Audit Act requires public accounting firms 
that are appointed auditors of certain agencies of the Crown to perform their audits under 
the direction of, and to report to, the Provincial Auditor. Under subsection 9(3) of the Act, 


public accounting firms auditing Crown-controlled corporations are required to deliver to 
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the Provincial Auditor a copy of the audited financial statements of the corporation and a 
copy of their report of their findings and recommendations to management (management 
letter). 


ADDITIONAL RESPONSIBILITIES 


Under section 16 of the Audit Act, the Provincial Auditor may, by resolution of the Standing 
Committee on Public Accounts, be required to examine and report on any matter 
respecting the Public Accounts. 


Section 17 of the Act requires the Provincial Auditor to undertake special assignments 
requested by the Assembly, the Standing Committee on Public Accounts (by resolution of 
the Committee), or by a minister of the Crown. However, these special assignments are not 
to take precedence over the Provincial Auditor’s other duties. The Provincial Auditor can 
decline an assignment referred by a minister if, in the opinion of the Provincial Auditor, it 
conflicts with other duties. 


AUDIT ACTIVITIES 
TYPES OF AUDITS 


Value-for-money, attest, and compliance audits are the three main types of audits carried 
out by the Office. The Office generally conducts compliance audit work as a component of 
its value-for-money and attest audits. In addition, inspection audits of selected grant- 
recipient organizations may be conducted under section 13 of the Audit Act. The following 
are brief descriptions of each of these audit types. 


Value-for-Money Audits 
Subclauses 12(2)(f)(iv) and (v) of the Audit Act require the Provincial Auditor to report on 


any cases observed where money was spent without due regard for economy and efficiency, 
or where appropriate procedures were not in place to measure and report on the 
effectiveness of programs. This value-for-money mandate is exercised through the auditing 
of various ministry and Crown-agency programs and activities each year. We have 
summarized in Chapter Three the conclusions, observations, and recommendations arising 
from the value-for-money work we performed between October 2002 and September 
2003. 


It is not part of the Office’s mandate to measure, evaluate, or report on the effectiveness of 
programs or to develop performance measures or standards. These functions are the 
responsibility of ministry and/or agency management. The Office is responsible for 
reporting instances noted where the ministry or agency has not carried out these functions 
satisfactorily. Our value-for-money work deals with the administration of programs by 
management, including major information systems. 
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We plan, perform, and report our value-for-money work in accordance with the 
professional standards for assurance engagements, encompassing value for money and 
compliance, established by the Canadian Institute of Chartered Accountants. These 
standards require that we employ rigorous processes to maintain the quality, integrity, and 
value of our work for our client, the Legislative Assembly. They also require that we clearly 
explain the nature and extent of the assurance provided as a result of our work. Some of 
these processes and the degree of assurance they enable us to provide are described below. 


SELECTION OF PROGRAMS AND ACTIVITIES FOR AUDIT 


Major ministry and agency programs and activities are audited at approximately five-year 
intervals. Various factors are considered in selecting programs and activities for audit each 
year. These factors include: the results of previous audits; the total revenues or expenditures 
at risk; the impact of the program or activity on the public; the inherent risk due to the 
complexity and diversity of operations; the significance of possible issues that may be 
identified by an audit; and the costs of performing the audit in relation to the perceived 
benefits. Possible issues are identified primarily through a preliminary survey of the program 
or agency. 

We also consider the work completed or planned by ministry and agency internal auditors. 
The relevance, timeliness, and breadth of scope of the work done by internal auditors can 
have a major impact on the timing, frequency, and extent of our audits. By having access to 
internal audit work plans, working papers, and reports and by relying, to the extent possible, 
on internal audit activities, the Office is able to avoid duplication of effort. 


OBJECTIVES AND ASSURANCE LEVELS 


The objective of our value-for-money work is to meet the requirements of subclauses 
12(2)(f) (iv) and (v) of the Audit Act by identifying and reporting significant value-for- 
money issues. We also include in our reports recommendations for improving controls, 
obtaining better value for money, and achieving legislated objectives. Management 
responses to our recommendations are reproduced in our reports. 


The specific objective(s) for each audit or review conducted are clearly stated in the 
Objective(s) and Scope section of each audit report—that is, each value-for-money section 
of Chapter Three. Our work is designed to allow us to conclude on our stated objective(s). 


In almost all cases, our work is planned and performed to provide an audit level of 
assurance. An audit level of assurance is obtained by: interviewing management and 
analyzing the information they provide; examining and testing systems, procedures, and 
transactions; confirming facts with independent sources; and, where necessary, obtaining 
expert assistance and advice in highly technical areas. 


An audit level of assurance refers to the highest reasonable level of assurance the Office can 
provide concerning the subject matter. Absolute assurance that all significant matters have 
been identified is not attainable for various reasons, including: the use of testing; the 
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inherent limitations of control; the fact that much of the evidence available is persuasive 
rather than conclusive in nature; and the need to exercise professional judgment. 


Infrequently, for reasons such as the nature of the program or activity, limitations in the 
Audit Act, or the prohibitive cost of providing a high level of assurance, the Office will 
perform a review rather than an audit. A review provides a moderate level of assurance 
because it consists primarily of inquiries and discussions with management; analyses of 
information they provide; and only limited examination and testing of systems, procedures, 
and transactions. For example, our work reported in Chapter Four of this report—Follow- 
up of Recommendations in the 2001 Annual Report—falls into this review category. 


CRITERIA 


In accordance with professional standards for assurance engagements, work is planned and 
performed to provide a conclusion on the objective(s) set for the work. A conclusion is 
reached and observations and recommendations made by evaluating the administration of a 
program or activity against suitable criteria. Suitable criteria are identified at the planning 
stage of our audit or review by performing extensive research of sources, such as: recognized 
bodies of experts; applicable laws, regulations, and other authorities; other bodies or 
jurisdictions delivering similar programs and services; management's own policies and 
procedures; and applicable criteria successfully applied in other audits or reviews. 


‘To further ensure their suitability, the criteria being applied are fully discussed with and 
agreed to by senior management responsible for the program or activity at the planning 
stage of the audit or review. 


COMMUNICATION WITH SENIOR MINISTRY OR AGENCY MANAGEMENT 


‘To help ensure the factual accuracy of our observations and conclusions, staff from our 
Office maintain ongoing communication with senior management throughout the audit or 
review. Before beginning the work, our staff meet with them to discuss the objectives and 
criteria and the focus of our work in general terms. During the audit or review, our staff 
meet with management to review progress and ensure open lines of communication. At the 
conclusion of on-site work, management is briefed on the preliminary results of the work. A 
draft report is then prepared and discussed with them. Management provides written 
responses to our recommendations and these are discussed and incorporated into the final 
draft report. The Provincial Auditor discusses and finalizes the draft report, on which the 
Annual Report will be based, with the deputy minister or agency head well in advance of 
the publication of the Annual Report. 


Attest Audits 


Attest (financial) audits are designed to permit the expression of the auditor’s opinion on a 
set of financial statements in accordance with generally accepted auditing standards. The 
opinion states whether the operations and financial position of the entity as reflected in their 
financial statements have been fairly presented in compliance with appropriate accounting 
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policies. The Office conducts attest audits of the financial statements of the province and of 
numerous Crown agencies on an annual basis. 


Compliance Audits 


Subsection 12(2) of the Audit Act also requires the Provincial Auditor to report observed 
instances noted where: 


* accounts were not properly kept or public money was not fully accounted for; 


* essential records were not maintained or the rules and procedures applied were not 
sufficient to safeguard and control public property or to effectively check the assessment, 
collection, and proper allocation of revenue or to ensure that expenditures were made 
only as authorized; or 


* money was expended other than for the purposes for which it was appropriated. 
Accordingly, as part of our value-for-money work, we: 


* identify provisions in legislation and authorities that govern the programs or agencies 
being examined or that the management of those programs or agencies is responsible for 
administering; and 


* perform such tests and procedures as we deem necessary to obtain reasonable assurance 
that management has complied with legislation and authorities in all significant respects. 


Inspection Audits of Grant-recipient Organizations 


Although grants to organizations such as hospitals, universities, community colleges, school 
boards, and thousands of smaller organizations amount to approximately 50% of total 
government expenditures, our Office can conduct only limited-scope inspection audits on 
them. Inspection audits are defined in the Audit Act as an examination of accounting 
records. Although value-for-money observations may arise as a by-product of inspection 
audits, the audits are not value-for-money oriented because only accounting records can be 
examined in inspection audits. 


The Office may, where circumstances warrant the extension of a ministry or agency audit, 
conduct inspection audits of grant recipients. In the past, the Office has carried out 
inspection audits of major recipients of grants, specifically community colleges, universities, 
hospitals, and school boards. However, in recent years, the Office has deferred major 
inspection-audit activity pending consideration of a proposal to amend the Audit Act to 
permit the Office to access all records and information necessary to perform full-scope 
audits, including value-for-money, of grant-recipient organizations. 


On April 19, 2001, the government announced in the Speech from the Throne, under the 
heading of “Holding the Broad Public Sector Accountable to Taxpayers,” that it would be 
introducing sweeping reforms to ensure that all public-sector institutions are accountable to 
the citizens of Ontario. Included in the planned reforms announced in the Throne Speech 
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was a commitment to make amendments to the Audit Act that would permit the Provincial 
Auditor to assess the extent to which institutions funded by Ontario taxpayers use that 
money prudently, effectively, and as intended. However, no action was taken by the 
government on this commitment. Further details and background on the subject of 
amendments to the Audit Act are provided in Chapter Two, in the section entitled 
“Proposed Amendments to the Audit Act.” 


Payments are also made to individuals under a variety of programs, such as the Ontario 
Health Insurance Plan or the Ontario Disability Support program. Such individual 
recipients of government funds are not, and should not be, subject to direct audit by the 
Provincial Auditor. For these kinds of programs, our audits focus on the ministries’ 
procedures to ensure that only eligible recipients are being paid the correct amount. 


REPORTING ACTIVITIES 
VALUE-FOR-MONEY AUDITS 


As each audit or review is completed, the Office prepares a preliminary draft report for 
discussion and factual clearance. The preliminary draft report is discussed with senior 
ministry or agency officials and revised, as necessary, to reflect the results of the discussion. 
The Provincial Auditor finalizes the draft report with the deputy minister or agency head 
(chair) well in advance of the publication of the Annual Report. Following clearance of the 
preliminary draft report and the ministry or agency response, a final draft report is prepared 
and issued to the deputy minister or agency head. These final draft audit reports form a 
significant component of our Annual Report to the Legislative Assembly. 


AGENCY ATTEST AUDITS 


With respect to reporting on attest audits of agencies, agency legislation normally stipulates 
that the Provincial Auditor’s reporting responsibilities are to the agency’s board and the 
minister(s) responsible. Also, we provide copies of the audit opinions and of the related 
agency financial statements to the deputy minister of the associated ministry, as well as to the 
Secretary of the Management Board of Cabinet. 


In instances where matters that require improvements by management have been noted 
during the course of an agency attest audit, a draft management letter is prepared, discussed 
with senior management, and revised as necessary to reflect the results of the discussion. 
Following clearance of the draft management letter and the response of the agency’s senior 
management, a final management letter is prepared and, if deemed necessary, issued to the 
agency head. Depending on the significance of the content of the management letter, a 
copy of it may also be forwarded to the minister and deputy minister of the associated 
ministry and to the Secretary of the Management Board of Cabinet. Matters of significance 
contained in the management letter may also be included in the Provincial Auditor’s 


Annual Report to the Legislative Assembly. 
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CONFIDENTIALITY OF WORKING PAPERS 


In the course of our reporting activities, we prepare draft audit reports and management 
letters that are considered to be an integral part of our audit working papers. It should be 
noted that these working papers, according to section 19 of the Audit Act, are not required 
to be laid before the Assembly or any of its committees. As well, because our Office is 
exempt from the Freedom of Information and Protection of Privacy Act, our reports and 
audit working papers, which include all information obtained from ministries and agencies 
during the course of an audit, may not be accessed from our Office, thus further ensuring 


confidentiality. 


SPECIAL ASSIGNMENTS 


Under sections 16 and 17 of the Audit Act, the Provincial Auditor has additional reporting 
responsibilities relating to special assignments for the Legislative Assembly, the Standing 
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Committee on Public Accounts, or a minister of the Crown. At the conclusion of such 
work, the Provincial Auditor normally reports to the initiating authority of the assignment. 


ANNUAL REPORT 


The reporting requirements for the Annual Report, as specified in subsection 12(2) of the 
Audit Act, are organized in the following way: 


Chapter One of the Annual Report provides an overview of the Provincial Auditor's 
findings for this year’s value-for-money audit activities as well as summaries of the value-for- 
money audits and reviews that were conducted. 


Chapter Two contains observations on the subject of improving public-sector accountability. 


Chapter Three contains the reports resulting from our value-for-money audits of ministries 
and agencies conducted during the year. 


Chapter Four contains the results of our follow-up on the action taken by each ministry and 
agency on the recommendations we made to them two years ago. We conduct this follow- 
up to encourage timely attention by each ministry and agency to our observations and 
recommendations. Accordingly, a detailed account of the current status of 
recommendations made in the 2001 Annual Report is provided in this chapter. 


Chapter Five is devoted to the Provincial Auditor's comments on the audit of the Public 
Accounts of the province. The reporting requirements under clauses 12(2)(d) and (e) of the 
Audit Act are also met in this chapter. 


In Chapter Six, we report on the activities of the Office of the Provincial Auditor and 
reproduce the Office’s externally audited statement of expenditure for the year ended 
March 31, 2003. 


Chapter Seven provides information on the composition and activities of the Standing 
Committee on Public Accounts. 
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OFFICE ORGANIZATION AND 
PERSONNEL 


The Office is organized into portfolio teams—a framework that attempts to align related 
audit entities and to foster expertise in the various areas of audit activity. The portfolios, 
which are loosely based on the government’s own ministry organization, are each headed by 
a Director who oversees and is responsible for the ministry and agency audits within the 
assigned portfolio. Assisting the portfolio Directors and rounding out the teams are a 
number of audit Managers and various other audit staff. 


The Provincial Auditor, the Assistant Provincial Auditor, and the portfolio Directors make 
up the Office’s Executive Steering Committee (ESC). The executive management of the 
Office as at September 30, 2003 is outlined in the following table. 


Executive Management of the Office of the Provincial Auditor, 2003 


Erik Peters Provincial Auditor (retired effective September 30, 2003) 
Jim McCarter Assistant Provincial Auditor 


Paul Amodeo Public Accounts; Finance; Information Technology; and 
Accounting Research 


Walter Bordne Community, Family and Children’s Services; and Revenue 
Andrew Cheung Justice and Regulatory 
Gerard Fitzmaurice Economic Development 


John McDowell Crown Agencies, Corporations, Boards, and 
Commissions; and Transportation and Energy 





Nick Mishchenko Health and Long-Term Care; and Management Board 
Secretariat 


Gary Peall Education; Training, Colleges and Universities; Culture: 
Municipal Affairs and Housing; and Professional 
Practices 








Note: Annemarie Wiebe, the Manager of Human Resources, regularly attends meetings of the 
ESC to provide advice on matters related to human resources. 


CODE OF PROFESSIONAL CONDUCT 


The Office has a Code of Professional Conduct to encourage staff to maintain high 
professional standards and ensure a professional work environment. It is intended to be a 
general statement of philosophy, principles, and rules regarding conduct for employees of 
the Office, who have a duty to conduct themselves in a professional manner and to strive to 
achieve the highest standards of behaviour, competence, and integrity in their work. The 
Code provides the reasoning for these expectations and further describes the Office’s 
responsibilities to the Legislative Assembly, the public, and our audit entities. The Code also 
provides guidance on disclosure requirements and the steps to be taken to avoid conflict-of- 
interest situations. 
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CANADIAN COUNCIL 
OF LEGISLATIVE AUDITORS 


The 31st annual meeting of the Canadian Council of Legislative Auditors (CCOLA) was 
held in Winnipeg, Manitoba, from September 14 to 16, 2003. This annual gathering, 
bringing together legislative auditors from the federal government and the provinces, 
provides a useful forum for sharing ideas and exchanging information important to the 
work of the legislative auditing community. 


The Provincial Auditor and the Assistant Provincial Auditor attended this year’s meeting, 
which covered such topics as: 


¢ Reporting Principles and Parliamentary Effectiveness 
¢ Performance Reporting 


¢ Reporting on Controls 


ACKNOWLEDGMENT 
AUDITEES AND STAFF 


The Provincial Auditor expresses sincere appreciation to those officials of ministries, 
agencies, and other entities for their co-operation in providing his staff with the information 
and explanations required during the performance of the Office’s audit work on a timely 
basis. However, there were two cases where we encountered difficulties in accessing 
information. In accordance with clause 12(2)(a) of the Audit Act, these cases are reported in 
Chapter Three, sections 3.01 and 3.07, and are summarized in Chapter One in the section 
entitled “Access to Information.” 


The Provincial Auditor extends a special appreciation to the staff of the Office for their 
dedication, competence, and the professional manner in which they have carried out their 
duties. 


OFFICE EXPENDITURE 


The following is the 2003 audited Statement of Expenditure for the Office. 
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Office of the Provincial Auditor of Ontario 
Statement of Expenditure 
Year Ended March 31, 2003 











2003 

‘Actual | Estimates | _ | Estimates — 

: | _ ($000) | ($000) | ($000) — 
Salaries and wages 5,364 5,822 4,721 5,822 
Employee benefits (note 3) 880 1135 716 878 
Transportation and communication 170 171 212 170 
Services 1,707 1,828 1,749 1,639 
Supplies and equipment 298 148 315 124 

Transfer payment: 

CCAF-FCVI Inc. 50 50 50 50 
The Audit Act (statutory) (note 4) 209 209 306 294 
: ee 8678  ## 9363 8069 — 8,977 


Notes to Statement of Expenditure: 


1. Accounting Policy 
The statement of expenditure has been prepared using the modified cash basis 
of accounting, which allows for an additional 30 days to pay for goods and 
services received during the fiscal year just ended. This basis of accounting is 
consistent with that used in the preparation of the Estimates and the prior year 
actual expenditures. 


2. Estimates 
The Estimates shown on the statement of expenditure reflect the approved 
Expenditure Estimates of the Province of Ontario for the respective years. 


3. Employee Benefits 
The Office provides pension benefits for eligible employees through participation 
in the Public Service Pension Fund (PSPF) and the Deputy Ministers’ 
Supplementary Benefit Plan, multi-employer pension plans established by the 
Province of Ontario. The plans are accounted for as defined contribution pension 
plans, as the Office has insufficient information to apply defined benefit plan 
accounting to these pension plans. Accordingly, the Office’s cash contributions 
to these plans for the year of $376,676 (2002—$190,869) reflect the Office’s 
pension expenditures and are included in employee benefits. 


The cost of post-retirement non-pension benefits is paid by Management Board 
Secretariat and therefore is not included in this statement. 


4. Statutory Appropriations 
Under section 5 of the Audit Act, the Provincial Auditor’s remuneration for the 
fiscal year is considered a statutory item. 
The actual 2002 expenditure includes, under section 23 oi'the Audit Act, 
specialist assistance of $97,370 incurred in connection with the Office’s 
examination of the Bruce Nuclear leasing agreement requested by the Standing 
Committee on Public Accounts and is considered a statutory item. 
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5. Public Sector Salary Disclosure Act, 1996 
This Act requires disclosure of Ontario public-sector employees paid an annual 
salary in excess of $100,000 in calendar year 2002. For the Office, this 
disclosure is as follows: 


Salary BE) c)e)(-) 
Paid Benefits 
$ 


$ 
Peters, Erik Provincial Auditor 209,490 8,826 
McCarter, James Assistant Provincial Auditor 149,445 
100,764 








Amodeo, Paul Director 
Bordne, Walter Director 110,957 
Cheung, Andrew Director 110,957 


Fitzmaurice, Gerard Director 113:545 























Peall, Gary 110,957 





McDowell, John Director 110,957 
Mishchenko, Nicholas Director 110,957 
Director 


AUDITORS’ REPORT TO THE BOARD OF INTERNAL ECONOMY 
OF THE LEGISLATIVE ASSEMBLY OF ONTARIO 


We have audited the statement of expenditure of the Office of the Provincial Auditor for 
the year ended March 31, 2003. This statement is the responsibility of the management of 
the Office of the Provincial Auditor. Our responsibility is to express an opinion on this 


statement based on our audit. 


We conducted our audit in accordance with Canadian generally accepted auditing 
standards. Those standards require that we plan and perform an audit to obtain reasonable 
assurance whether the statement is free of material misstatement. An audit includes 
examining, on a test basis, evidence supporting the amounts and disclosures in the 
statement. An audit also includes assessing the accounting principles used as well as 
evaluating the overall statement presentation. 


In our opinion, this statement presents fairly, in all material respects, the expenditures of the 
Office of the Provincial Auditor for the year ended March 31, 2003 in accordance with the 


accounting policy referred to in note | to the statement. 


Toronto, Canada ALLEN & MILES LLP 
September 4, 2003 CHARTERED ACCOUNTANTS 
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PROVINCIAL AUDITOR’S RETIREMENT 


On September 30, 2003, Erik Peters, FCA, retired from the Office after serving almost 
11 years as Provincial Auditor. 


Erik Peters served during a very important transition period in government accounting. He 
was instrumental in bringing about significant improvements in the province’s accounting 
and financial-reporting practices. Such changes included: the government’s move from a 
cash basis of accounting to accrual accounting; the government's consistent use of new 
accounting rules in preparing the Budget, the Estimates, and the financial statements; and 
the government’s more inclusive financial reporting of all its activities. During his tenure, 
Mr. Peters placed an equally high priority on accountability and the Office’s value-for- 


money mandate. 


Mr. Peters was the year 2000 recipient of the Institute of Chartered Accountants of 
Ontario's Award of Outstanding Merit for conspicuous and sustained service to the 
profession and the broader community. 
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CHAPTER SEVEN 


The Standing Committee 
on Public Accounts 





APPOINTMENT AND COMPOSITION OF 
THE COMMITTEE 


The Standing Orders of the Legislature provide for the appointment of an all-party 
Standing Committee on Public Accounts. The Committee is appointed for the duration of 
the Parliament (that is, the period from the opening of the first session immediately 
following a general election to the end of a government's term and the calling of another 
election). 


The membership of the Committee reflects proportionately the representation of parties in 
the Legislature. All members are entitled to vote on motions with the exception of the Chair, 
whose vote is restricted to the breaking of a tte. 


In accordance with the Standing Orders, a Standing Committee on Public Accounts was 
appointed on November 1, 1999, soon after the commencement of the First Session of the 
Thirty-seventh Parliament. The activities of the Committee effectively came to an end with 
the dissolution of the House on September 2, 2003. The membership of the Committee at 
dissolution was as follows: 


John Gerretsen, Chair, Liberal 

Bruce Crozier, Vice-chair, Liberal 

Garfield Dunlop, Progressive Conservative 
Raminder Gill, Progressive Conservative 
John Hastings, Progressive Conservative 
Shelley Martel, New Democrat 

Al McDonald, Progressive Conservative 
Richard Patten, Liberal 


ROLE OF THE COMMITTEE 


The Committee examines, assesses, and reports to the Legislature on a number of issues, 
including: the economy and efficiency of government operations; the effectiveness of 
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programs in achieving their objectives; controls over assets, expenditures, and the assessment 
and collection of revenues; and the reliability and appropriateness of information in the 
Public Accounts. 


In fulfilling this role, pursuant to its terms of reference in the Standing Orders of the 
Assembly, the Committee reviews the Provincial Auditor’s Annual Report and the Public 
Accounts and reports to the Legislature its observations, opinions, and recommendations. 
Under the Standing Orders, the documents are deemed to have been permanently referred 
to the Committee as they become available. 


PROVINCIAL AUDITOR’S ROLE WITH THE 
COMMITTEE 


In accordance with section 16 of the Audit Act, the Provincial Auditor and senior staff 
attend committee meetings at which the Committee reviews the Provincial Auditor’s Annual 
Report and the Public Accounts and assist the Committee in planning its agenda. 


COMMITTEE PROCEDURES AND 
OPERATIONS 


GENERAL 


The Committee meets on Thursday mornings when the Legislature is sitting. At times, the 
Committee also meets during the summer and winter when the Legislature is not sitting. All 
meetings are open to the public with the exception of those dealing with the setting of the 
Committee's agenda and the preparation of committee reports. All public Committee 
proceedings are recorded in Hansard (the official verbatim report of the debates in the 
House, speeches, other proceedings in the Legislature, and all open-session sittings of 
standing and select committees). 


The Committee selects matters from the Provincial Auditor's Annual Report and the Public 
Accounts for hearings. The Committee then requests senior ministry and agency officials to 
appear and respond to questions at these hearings. Since the Committee is concerned with 
administrative rather than policy matters, ministers rarely attend. 


Once hearings are completed, the Committee then reports its comments and 
recommendations to the Legislature. Committee procedures include the following: 


¢ in-depth briefings and preparation; 
* when practical, the inclusion of ministry responses in committee reports; and 


° follow-up of committee recommendations. 
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The Committee also follows up in writing with those ministries and Crown agencies not 
selected for detailed review by the Committee regarding their plans and timetables for 
addressing the concerns raised in the Provincial Auditor’s reports. This process enables each 
auditee to update the Committee on activities undertaken since the completion of the audit, 
such as any initiatives taken to address the Provincial Auditor’s recommendations. 


MEETINGS HELD 


From October 2002 to September 2003, the Committee met regularly on Thursday 
mornings when the Legislature was sitting and also met during the winter recess to consider 
the reports of the Provincial Auditor. The Committee’s work during this period relating to 


the work of this Office included: 
¢ considering Bill 5, Audit Statute Law Amendment Act, 2002; 
¢ reviewing the following items from the Provincial Auditor’s 2002 Annual Report: 


- Ministry of Finance—Corporations Tax Program and the Public Accounts of the 
province; 


- Ministry of Health and Long-Term Care—Long-Term Care Facilities Activity; 


- Management Board Secretariat and the ministries of Environment, Finance, Health 
and Long-Term Care, Natural Resources, and Public Safety and Security— 
Consulting Services; 


- Ministry of Public Safety and Security—Community Services Program; 
- Ministry of Tourism and Recreation—Tourism Program; 
- Ministry of Training, Colleges and Universities— Training Division; and 


- the following follow-ups of recommendations contained in the 2000 Special Report 
on Accountability and Value for Money: 


2 Ministry of Public Safety and Security—Institutional Services and Young 
Offender Operations; and 


2 Ministry of Natural Resources—Forest Management; and 


¢ finalizing reports to the Legislature. 


REPORTS OF THE COMMITTEE 
GENERAL 


The Committee issues its reports to the Legislature. These reports contain a précis of the 
information reviewed by the Committee during its meetings, together with comments and 


recommendations. 
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All committee reports are available through the Clerk of the Committee, thus enabling 
public access to full details of committee deliberations. 


After the Committee tables its report in the Legislative Assembly, it requests that ministries 
or agencies respond within 120 days or within time frames stipulated by the Committee in 
its individual recommendations. 


During the period from October 2002 to September 2003, the Committee submitted the 
following reports to the Legislative Assembly: 


¢ keport on Ontario Innovation Trust; 

¢ keport on the Road User Safety Program; 

¢ keport on the Integrated Justice Project, 

¢ Report on the Food Industry Program; 

° keport on Violence Against Women Program; 
° Report on Community Reinvestment Fund; 

¢ Report on Corporations Tax; 

°  keport on Training Division; 

° Report on Long-Term Care Facilities; and 


¢ Report on Consulting Services. 


FOLLOW-UP OF RECOMMENDATIONS MADE BY 
THE COMMITTEE 


The Clerk of the Committee is responsible for following up on the actions taken on the 
Committee’s recommendations by ministries or agencies. The Office of the Provincial 
Auditor confers with the Clerk to ascertain the status of the recommendations and, if 


considered necessary, brings any significant matters to the attention of the Legislature 
through the Provincial Auditor's Annual Report. 


Detailed information on the Committee’s recommendation respecting amendments to the 
Audit Act and other related matters is contained in Chapter Two of this report. 


OTHER COMMITTEE ACTIVITIES 


Canadian Council of Public Accounts Committees 
(CCPAC) 


CCPAC consists of delegates from federal, provincial, and territorial public accounts 
committees from across Canada. CCPAC meets at the same time and place as the Canadian 
Council of Legislative Auditors (CCOLA) to discuss issues of current interest. The 
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twenty-fourth annual meeting of CCPAC was held in Winnipeg, Manitoba from 
September 14 to 16, 2003. The annual CCPAC and CCOLA meetings also permit the 
delegations to participate in a joint session to discuss subjects of mutual interest to politicians 
and legislative auditors. The 2003 joint session with CCOLA was on the subject of 
“Reporting Principles and Parliamentary Effectiveness.” Due to the dissolution of the 
Ontario Legislature on September 2, 2003, it was not possible for Ontario to be 
represented at the 2003 meeting of CCPAC. 
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EXHIBIT ONE 


Agencies of the Crown 


(I) Agencies whose accounts are audited by the Provincial Auditor 


AgriCorp 

Algonquin Forestry Authority 

Cancer Care Ontario 

Centennial Centre of Science and Technology 

Chief Election Officer, Election Finances Act — 

Election Fees and Expenses, Election Act 

Financial Services Commission of Ontario 

Grain Financial Protection Board, Funds for Producers of Grain Corn, Soybeans and Canola 
Investor Education Fund, Ontario Securities Commission 
Legal Aid Ontario 

Liquor Control Board of Ontario 

Livestock Financial Protection Board, Fund for Livestock Producers 
Northern Ontario Heritage Fund Corporation 

North Pickering Development Corporation 

Office of the Assembly 

Office of the Environmental Commissioner 

Office of the Information and Privacy Commissioner 
Office of the Children’s Lawyer 

Office of the Ombudsman 

Ontario Clean Water Agency (December 31) 

Ontario Development Corporation 

Ontario Educational Communications Authority 

Ontario Electricity Financial Corporation 

Ontario Exports Inc. 

Ontario Financing Authority 

Ontario Food Terminal Board 

Ontario Heritage Foundation 

Ontario Housing Corporation (December 31) 

Ontario Immigrant Investor Corporation 

Ontario Media Development Corporation 

Ontario Municipal Economic Infrastructure Financing Authority 


Ontario Northland Transportation Commission (December 31) 


Ontario Place Corporation 

Ontario Racing Commission 

Ontario Realty Corporation 

Ontario Securities Commission 

Ontario SuperBuild Corporation 

Ontario Tourism Marketing Partnership Corporation 
Owen Sound Transportation Company Limited 
Province of Ontario Council for the Arts 

Provincial Judges Pension Fund, Provincial Judges Pension Board 
Public Guardian and Trustee for the Province of Ontario 
Toronto Area Transit Operating Authority 

TVOntario Foundation 
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(Il) Agencies whose accounts are audited by another auditor 


under the direction of the Provincial Auditor 






Board of Community Mental Health Clinic, Guelph 
Motor Vehicle Accident Claims Fund 

Niagara Parks Commission (October 31) 

Ontario Mental Health Foundation 

St. Lawrence Parks Commission 


Workplace Safety and Insurance Board (December 31) 










Notes: 
1. Dates in parentheses indicate fiscal periods ending on a date other than March 31. 


2. Changes during the 2002/03 fiscal year: 


Additions: 
* Motor Vehicle Accident Claims Fund 
* — Ontario Municipal Economic Infrastructure Financing Authority 


* Owen Sound Transportation Company Limited 


——— eee 


394 2003 Annual Report of the Office of the Provincial Auditor of Ontario 





EXHIBIT TWO 


Crown-controlled 
Corporations 





Corporations whose accounts are audited by an auditor other than the Provincial 
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Access Centre for Community Care in Lanark, Leeds and Grenville 
Access Centre for Hastings and Prince Edward Counties 
Algoma Community Care Access Centre 







Art Gallery of Ontario Crown Foundation 






Baycrest Hospital Crown Foundation 






Board of Funeral Services 

Brant Community Care Access Centre 

Brock University Foundation 

Carleton University Foundation 

CIAR Foundation (Canadian Institute for Advanced Research) 
Canadian Opera Company Crown Foundation 








Canadian Stage Company Crown Foundation 

Chatham/Kent Community Care Access Centre 

Cochrane District Community Care Access Centre 

Community Care Access Centre (CCAC) — Oxford 

Community Care Access Centre for Huron 

Community Care Access Centre for Kenora and Rainy River Districts 
Community Care Access Centre for the Eastern Counties 
Community Care Access Centre Niagara 

Community Care Access Centre of Halton 

Community Care Access Centre of London and Middlesex 














Community Care Access Centre of Peel 
Community Care Access Centre of Waterloo Region 
Community Care Access Centre Wellington-Dufferin 
Community Care Access Centre of York Region 

Community Care Access Centre Perth County 

Community Care Access Centre Simcoe County 

Community Care Access Centre Timiskaming 

Community Care Access Centre of The District of Thunder Bay 
Deposit Insurance Corporation of Ontario 

Durham Access to Care 

East York Access Centre for Community Services 

Education Quality and Accountability Office 

Elgin Community Care Access Centre 
Etobicoke Community Care Access Centre 
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_ Corporations whose accounts are audited by an auditor other than the Provincial 
Auditor, with full access by the Provincial Auditor to audit reports, working papers, 


_ and other related documents 
(eyalitattc-ye)) 








Foundation at Queen’s University at Kingston 
GO Transit 

Grand River Hospital Crown Foundation 
Grey-Bruce Community Care Access Centre 
Haldimand-Norfolk Community Care Access Centre 

Haliburton, Northumberland and Victoria Long-Term Care Access Centre 
Hamilton Community Care Access Centre 

Hydro One Inc. 

Kingston, Frontenac, Lennox and Addington Community Care Access Centre 
Lakehead University Foundation 

Laurentian University of Sudbury Foundation 

Manitoulin-Sudbury Community Care Access Centre 

McMaster University Foundation 

McMichael Canadian Art Collection 

Metropolitan Toronto Convention Centre Corporation 

Mount Sinai Hospital Crown Foundation 

National Ballet of Canada Crown Foundation 

Near North Community Care Access Centre 

Nipissing University Foundation 

North York Community Care Access Centre 

North York General Hospital Crown Foundation 

Ontario Family Health Network 

Ontario Foundation for the Arts 

Ontario Lottery and Gaming Corporation 

Ontario Mortgage Corporation 

Ontario Municipal Employees Retirement Board 































Ontario Pension Board 







Ontario Power Generation Inc. 






Ontario Trillium Foundation 
Ottawa Community Care Access Centre 






Ottawa Congress Centre 
Renfrew County Community Care Access Centre 
Royal Botanical Gardens Crown Foundation 








Royal Ontario Museum 






Royal Ontario Museum Crown Foundation 
Ryerson Polytechnic University Foundation 
Sarnia/Lambton Community Care Access Centre 








Scarborough Community Care Access Centre 
Science North 
Shaw Festival Crown Foundation 







Smart Systems for Health Agency 






St. Clair Parks Commission 

St. Michael’s Hospital Crown Foundation 
Stadium Corporation of Ontario Limited 
Stratford Festival Crown Foundation 









Sunnybrook Hospital Crown Foundation 
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The Peterborough Community Access Centre Incorporated 


Toronto Community Care Access Centre 

Toronto East General Hospital Crown Foundation 
Toronto Hospital Crown Foundation 

Toronto Islands Residential Community Trust Corporation 
Toronto Symphony Orchestra Crown Foundation 

Trent University Foundation 

University of Guelph Foundation 

University of Ottawa Foundation 

University of Toronto Foundation 

University of Waterloo Foundation 

University of Western Ontario Foundation 

University of Windsor Foundation 

Waterfront Regeneration Trust Agency 

Wilfrid Laurier University Foundation 

Windsor/Essex Community Care Access Centre 

Women’s College and Wellesley Central Crown Foundation 
York Community Care Access Centre 

York University Foundation 





Notes: 
Changes during the 2002/03 fiscal year: 
Deletions: 

* Big Thunder Sports Park Ltd. 

« Dairy Farmers of Ontario 
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EXHIBIT THREE 


Treasury Board Orders 





Under sub-section 12(2)(e) of the Audit Act, the Provincial Auditor is required to annually 
report all orders of the Management Board of Cabinet authorizing payments in excess of 
appropriations, stating the date of each order, the amount authorized, and the amount 
expended. These are outlined in the following table. 


Amounts Authorized and Expended Thereunder Year Ended March 31, 2003 










Agriculture and Food Oct. 31, 2002 52,000,000 52,000,000 







Feb. 27, 2003 33,349,200 32,850,270 
Apr. 17, 2003 3,040,100 <= 
88,389,300 84,850,270 









Attorney General Feb. 27, 2003 57,355,300 50,390,240 












Apr. 3, 2003 1,489,200 1,489,152 
Apr. 24, 2003 700,000 697,432 
59,544,500 52,576,824 














Cabinet office Apr. 17, 2003 452,000 412,543 

















Citizenship Apr. 17, 2003 2,023,800 1,613,539 















Community, Family and Children’s 












Services Feb. 13, 2003 18,506,300 18,506,300 
Mar. 27, 2003 35,320,000 24,721,647 
53,826,300 43,227,947 

















Consumer and Business Services Feb. 13, 2003 9,448 400 8,136,585 
Feb. 26, 2003 200,000 200,000 
9,648,400 8,336,585 


























Culture Feb. 20, 2003 1,000,000 997,000 
Apr. 17, 2003 2,199,000 2,048,939 
3,199,000 3,045,939 























Education Mar. 6, 2003 262,713,600 258,195,711 






Mar. 27, 2003 51,300,000 28,778,504 
314,013,600 286,974,215 












Enterprise, Opportunity and 
Innovation Apr. 3, 2003 685,000 519,467 
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Ministry 


Environment and Energy 


Finance 


Office of Francophone Affairs 


Health and Long-Term Care 


Labour 
Office of the Lieutenant Governor 


Management Board Secretariat 


Municipal Affairs and Housing 


Natural Resources 


Northern Development and Mines 
Office of the Premier 


Public Safety and Security 





Treasury Board Orders 


By \ {We} @) ge(-1¢ 


Feb. 6, 2003 

Feb. 26, 2003 
Feb. 26, 2003 
Mar. 27, 2003 


Jan. 16, 2003 
Jan. 23, 2003 
Feb. 27, 2003 


Dec. 12, 2002 
Apr. 17, 2003 


Feb. 13, 2003 
Mar. 20, 2003 
Apr. 3, 2003 


Mar. 20, 2003 
Nov. 28, 2002 


Oct. 3, 2002 
Mar. 6, 2003 


June 27, 2002 
Aug, 13, 2002 
Sept. 12, 2002 
Mar. 27, 2003 
Mar. 27, 2003 
Mar. 27, 2003 
Mar. 27, 2003 


June 20, 2002 
Aug. 8, 2002 
Feb. 13, 2003 
Mar. 6, 2003 
Mar. 20, 2003 


Mar. 20, 2003 
Feb. 27, 2003 


Aug. 8, 2002 

Dec. 12, 2002 
Feb. 13, 2003 
Mar. 20, 2003 
Apr. 17, 2003 


PNT inte) ayd-ce| 
$ 


2,500,000 

500,000 
1,440,000 
3,828,400 





8,268,400 


52,147,400 
30,385,800 
7,345,800 


=> 4 el-1alo(-1e| 
$ 


2,470,204 

385,220 
1,440,000 
2,526,311 


6,821,735 


52,147,400 
16,351,024 
411,531 





89,879,000 


68,909,955 





37,500 
57,400 
94,900 


59,000,000 
155,728,300 
33,469,200 





58,620,532 
140,472,237 
23,716,163 








248,197,500 


1,797,300 


222,808,932 


900,258 





94,800 


56,239,500 
23,799,000 


80,038,500 


93,584 


44,966,780 
22,001,149 


66,967,929 





1,250,000 
2,500,000 
1,586,000 
496,700 
11,000,000 
1,140,000 
4,152,000 


1,250,000 
2,206,066 
1,586,000 
496,700 
10,997,487 


1,792,448 





22,124,700 


7,500,000 
30,000,000 
4,634,300 
4,898,600 
9,936,100 


18,328,701 


7,500,000 
30,000,000 
4,451,296 
4,898,600 
6,340,309 





56,969,000 


794,400 





736,200 


53,190,205 


602,051 


Feo.ore 





29,071,100 
86,482,400 
89,386,800 
52,099,700 

283,600 





257,323,600 


29,071,100 
86,482,400 
87,170,944 
45,413,633 

224,321 


248 362,398 
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Ministry 
Tourism and Recreation 


Training, Colleges and Universities 


Transportation 


ree 
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Date of Order 


Apr. 17, 2003 


Feb. 12, 2003 
Feb. 12, 2003 
Mar. 6, 2003 


Feb. 27, 2003 
Mar. 27, 2003 
Apr. 3, 2003 


Authorized 


$ 


2,169,000 


4,200,000 
315,600 
53,321,200 


57,836,800 


1,596,000 
22,677,000 
10,740,000 


35,013,000 


Expended 
$ 


740,502 
3,500,000 


315,600 
50,984,665 


54,800,265 


1,583,367 
21,064,861 


9,926,284 


32,574,512 











EXHIBIT FOUR 


The Audit Act 
R.S.O. 1990, Chapter A.35 





Amended by:1999% ch sale 999 %e0 11. 
Definitions 1. Inthis Act, 


“agency of the Crown” means an association, authority, board, commis- 
sion, corporation, council, foundation, institution, organization or 


other body, 


(a) whose accounts the Auditor is appointed to audit by its sharehold- 
ers or by its board of management, board of directors or other 
governing body, 


(b) whose accounts are audited by the Auditor under any other Act or 
whose accounts the Auditor is appointed by the Lieutenant Gover- 
nor in Council to audit, 


(c) whose accounts are audited by an auditor, other than the Auditor, 
appointed by the Lieutenant Governor in Council, or 


(d) the audit of the accounts of which the Auditor is required to direct 
or review or in respect of which the auditor's report and the 
working papers used in the preparation of the auditor's statement 
are required to be made available to the Auditor under any other 
Act, 


but does not include one that the Crown Agency Act states is not 
affected by that Act or that any other Act states is not a Crown agency 
within the meaning or for the purposes of the Crown Agency Act; 
(“organisme de la Couronne”) 


“Assistant Auditor’ means the Assistant Provincial Auditor; (“Vérificateur 
adjoint”) 
“Auditor” means the Provincial Auditor; (“Vérificateur”) 


“Board” means the Board of Internal Economy referred to in section 87 


of the Legislative Assembly Act; (“Commission”) 
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Office of the Auditor 


Provincial Auditor 


Tenure of office and 
removal 


Salary of Auditor 


Same 


Idem 


“Crown controlled corporation” means a corporation that is not an 
agency of the Crown and having 50 per cent or more of its issued and 
outstanding shares vested in Her Majesty in right of Ontario or having 
the appointment of a majority of its board of directors made or ap- 
proved by the Lieutenant Governor in Council; (“société contrélée par 
la Couronne’) 


“fiscal year” has the same meaning as in the Ministry of Treasury and 
Economics Act, (“exercice”) 


“inspection audit” means an examination of accounting records; 
(“vérification’) 
“Office of the Auditor” means the Office of the Provincial Auditor; 


(“Bureau du Vérificateur’) 


“public money” has the same meaning as in the Financial Administration 


Act. ( deniets publics ) RS. ©. 1990 crASoyen 1: 
2. The Office of the Provincial Auditor shall consist of the Auditor, 


the Assistant Auditor and such employees as may be required from time 
to time for the proper conduct of the business of the Office. 
Res-OnlOOU Cr ons. 


3. The Auditor shall be appointed as an officer of the Assembly by the 
Lieutenant Governor in Council on the address of the Assembly after 
consultation with the chair of the standing Public Accounts Committee 


of the Assembly. R.S.O. 1990, c. A.35, s. 3. 


4. The Auditor may hold office until the end of the month in which 
he or she attains the age of sixty-five years and may be reappointed for a 
period not exceeding one year at a time until the end of the month in 
which he or she attains seventy years of age, but is removable at any time 
for cause by the Lieutenant Governor in Council on the address of the 


Assembly. R.S.O. 1990, c. A.35, s. 4. 
5.—(1) The Auditor shall be paid a salary within the highest range of 


salaries paid to deputy ministers in the Ontario civil service and is entitled 
to the privileges of office of a senior deputy minister. R.S.O. 1990, 
CoALD 53S: OL) IOI ere oe ltGl)s| 999s n lam c wlan 


(2) The salary of the Auditor, within the salary range referred to in 
subsection (1), shall be determined and reviewed ‘annually by the Board. 
L999 eel ess (2) 


(3) The salary of the Auditor shall be charged to and paid out of the 
Consolidated Revenue Fund. R.S.O. 1990, c. A.35,s. 5 (3). 
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The Audit Act 


6. The Assistant Auditor shall be appointed as an officer of the Assem- 
bly by the Lieutenant Governor in Council upon the recommendation 


of the Auditor. R.S.O. 1990, c. A.35, s. 6. 
7. The Assistant Auditor, under the direction of the Auditor, shall 


assist in the exercise of the powers and the performance of the duties of 
the Auditor and, in the absence or inability to act of the Auditor, shall act 
in the place of the Auditor. R.S.O. 1990, c. A.35, s. 7. 


8. The persons appointed as Auditor and Assistant Auditor shall be 
persons who are licensed under the Public Accountancy Act. 


ReS;051990-Gn 35, S20. 
9.—(1) The Auditor shall audit, on behalf of the Assembly and in 


such manner as the Auditor considers necessary, the accounts and 
records of the receipt and disbursement of public money forming part of 
the Consolidated Revenue Fund whether held in trust or otherwise. 


RES) 9905-c#A-35; 809. (1): 


(2) Where the accounts and financial transactions of an agency of the 
Crown are not audited by another auditor, the Auditor shall perform the 
audit, and, despite any other Act, where the accounts and financial 
transactions of an agency of the Crown are audited by another auditor, 
the audit shall be performed under the direction of the Auditor and such 
other auditor shall report to the Auditor. R.S.O. 1990, c. A.35,s. 9 (2). 


(3) Where the accounts of a Crown controlled corporation are 
audited other than by the Auditor, the person or persons performing the 
audit, 


(a) shall deliver to the Auditor forthwith after completion of the audit 
a copy of their report of their findings and their recommendations 
to the management and a copy of the audited financial statements 
of the corporation; 


shall make available forthwith to the Auditor, when so requested 
by the Auditor, all working papers, reports, schedules and other 
documents in respect of the audit or in respect of any other audit 
of the corporation specified in the request; 


S 


(c) shall provide forthwith to the Auditor, when so requested by the 
Auditor, a full explanation of work performed, tests and 
examinations made and the results obtained, and any other 
information within the knowledge of such person or persons in 


respect of the corporation. R.S.O. 1990, c. A.35,s. 9 (3). 
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Additional 
examination and 
investigation 


Information and 
access to records 


Accommodation in 
ministries and 
Crown agencies 


Annual report 


Contents of report 


(4) Where the Auditor is of the opinion that any information, 
explanation or document that is provided, made available or delivered to 
him or her by the auditor or auditors referred to in subsection (2) or (3) 
is insufficient, the Auditor may conduct or cause to be conducted such 
additional examination and investigation of the records and operations of 
the agency or corporation as the Auditor considers necessary. 


Res Or 1990 7c oe 4). 


10. Every ministry of the public service, every agency of the Crown 
and every Crown controlled corporation shall furnish the Auditor with 
such information regarding its powers, duties, activities, organization, 
financial transactions and methods of business as the Auditor from time 
to time requires, and the Auditor shall be given access to all books, 
accounts, financial records, reports, files and all other papers, things or 
property belonging to or in use by the ministry, agency of the Crown or 
Crown controlled corporation and necessary to the performance of the 


duties of the Auditor under this Act. R.S.O. 1990, c. A.35,s. 10. 


11. For the purposes of exercising powers or performing duties under 
this Act, the Auditor may station one or more members of the Office of 
the Auditor in any ministry of the public service, in any agency of the 
Crown and in any Crown controlled corporation and the ministry, 
agency or corporation shall provide such accommodation as is required 


for such purposes. R.S.O. 1990, c. A.35,s. 11. 


12.—(1) The Auditor shall report annually to the Speaker of the 
Assembly after each fiscal year is closed and the Public Accounts are laid 
before the Assembly, but not later than the 31st day of December in each 
year unless the Public Accounts are not laid before the Assembly by that 
day, and may make a special report to the Speaker at any time on any 
matter that in the opinion of the Auditor should not be deferred until 
the annual report, and the Speaker shall lay each such report before the 
Assembly forthwith if it is in session or, ifnot, not later than the tenth day 
of the next session. R.S.O. 1990, c. A.35,s. 12 (1). 


(2) In the annual report in respect of each fiscal year, the Auditor shall 


report on, 


(a) the work of the Office of the Auditor, and on whether in carrying 
on the work of the Office the Auditor received all the information 
and explanations required; 


(b) the examination of accounts of receipts and disbursements of 
public money; 
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the examination of the statements of Assets and Liabilities, the 
Consolidated Revenue Fund and Revenue and Expenditure as 
reported in the Public Accounts, and shall express an opinion as to 
whether the statements present fairly the financial position of the 
Province, the results of its operations and the changes in its 
financial position in accordance with the accounting principles 
stated in the Public Accounts applied on a basis consistent with 
that of the preceding fiscal year together with any reservations the 
Auditor may have; 


all special warrants issued to authorize payments, stating the date of 
each special warrant, the amount authorized and the amount 


expended; 


all orders of the Management Board of Cabinet made to authorize 
payments in excess of appropriations, stating the date of each 
order, the amount authorized and the amount expended; 


such matters as, in the opinion of the Auditor, should be brought 
to the attention of the Assembly including, without limiting the 
generality of the foregoing, any matter related to the audit of 
agencies of the Crown or Crown controlled corporations or any 
cases where the Auditor has observed that, 


(i) accounts were not properly kept or public money was not 
fully accounted for, 


(ii) essential records were not maintained or the rules and 
procedures applied were not sufficient to safeguard and 
control public property or to effectively check the assessment, 
collection and proper allocation of revenue or to ensure that 
expenditures were made only as authorized, 


(iii) money was expended other than for the purposes for which it 
was appropriated, 


(iv) money was expended without due regard to economy and 
efficiency, or 


(v) where procedures could be used to measure and report on 
the effectiveness of programs, the procedures were not 
established or, in the opinion of the Auditor, the established 
procedures were not satistactory: Res 1990) coA.5), 

a) PAPAL 
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Inspection audit 13.—(1) The Auditor may perform an inspection audit in respect ofa 
payment in the form of a grant from the Consolidated Revenue Fund or 
an agency of the Crown and may require a recipient of such a payment 
to prepare and to submit to the Auditor a financial statement that sets out 
the details of the disposition of the payment by the recipient. 

RES-OR1 990 res Asoc oid): 


Obstruction of (2) No person shall obstruct the Auditor or any member of the Office 

Auditor of the Auditor in the performance of an inspection audit or conceal or 
destroy any books, papers, documents or things relevant to the subject- 
matter of the inspection audit. R.S.O. 1990, c. A.35,s. 13 (2). 





Offence (3) Every person who knowingly contravenes subsection (2) and every 
director or officer of a corporation who knowingly concurs in such 
contravention is guilty of an offence and on conviction is liable to a fine of 
not more than $2,000 or to imprisonment for a term of not more than 


one year, or to both: (R3S:O9199070) A355; S213 (3). 
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Idem, corporation (4) Where a corporation is convicted of an offence under subsection 
(3), the maximum penalty that may be imposed upon the corporation is 


$25,000 and not as provided therein. R.S.O. 1990, c. A.35, s. 13 (4). 


Examination on oath = 14, ‘The Auditor may examine any person on oath on any matter 
pertinent to any account subject to audit by the Auditor or in respect of 
any inspection audit by the Auditor and for the purpose of such an 
examination the Auditor has the powers conferred upon a commission 
under Part I of the Public Inquiries Act, which Part applies to the exami- 
nation as if it were an inquiry under that Act. R.S.O. 1990, c. A.35, 

s. 14. 


Proviso 15. Nothing in this Act shall be construed to require the Auditor, 


(a) to report on any matter that, in the opinion of the Auditor, is 
immaterial or insignificant; or 


(b) to audit or direct the audit of or report on the accounts of a body 
not referred to in this Act in the absence of such a requirement in 


any other Act in respect of the body. R.S.O. 1990, c. A.35, s. 15. 


Attendance at 16. At the request of the standing Public Accounts Committee of the 
standing Public Assembly, the Auditor and any member of the Office of the Auditor 


Accounts Committee ; : : ‘ 
ofthe Ae designated by the Auditor shall attend at the meetings of the committee 


in order, 
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(a) to assist the committee in planning the agenda for review by the 
committee of the Public Accounts and the annual report of the 


Auditor; and 


(b) to assist the committee during its review of the Public Accounts 
and the annual report of the Auditor, 


and the Auditor shall examine into and report on any matter referred to 
him or her in respect of the Public Accounts by a resolution of the 


committee. R.S.O. 1990, c. A.35,s. 16. 


Special assignments 17. The Auditor shall perform such special assignments as may be 
required by the Assembly, the standing Public Accounts Committee of 
the Assembly, by resolution of the committee, or by a minister of the 
Crown in right of Ontario but such special assignments shall not take 
precedence over the other duties of the Auditor under this Act and the 
Auditor may decline an assignment by a minister of the Crown that, in 
the opinion of the Auditor, might conflict with the other duties of the 
Aniditor B.S OL), cao, sk 


Power to advise 18. The Auditor may advise appropriate persons employed in the 
public service of Ontario as to any matter that comes or that may come to 
the attention of the Auditor in the course of exercising the powers or 


performing the duties of Auditor. R.S.O. 1990, c. A.35,s. 18. 


Audit working 19. Audit working papers of the Office of the Auditor shall not be 
a rh laid before the Assembly or any committee of the Assembly. 
ReSi@ 1990 Gc. A 3s s.5ho. 


Staff 20. Subject to the approval of the Board and to sections 22, 25 and 
26, the Auditor may employ such professional staffand other persons as 
the Auditor considers necessary for the efficient operation of the Office 
of the Auditor and may determine the salary of the Assistant Auditor and 
the salaries and remuneration, which shall be comparable to the salary 
ranges of similar positions or classifications in the public service of On- 
tario, and the terms and conditions of employment of the employees of 


the Office of the Auditor. R.S.O. 1990, c. A.35,s. 20. 


Oath of office and 21.—(1) Every employee of the Office of the Auditor, before per- 
secrecy and oath of forming any duty as an employee of the Auditor, shall take and subscribe 
oe before the Auditor or a person designated in writing by the Auditor, 


(a) the following oath of office and secrecy, in English or in French: 


| REE OR AR nena do swear (or solemnly affirm) that I will faithfully 
discharge my duties as an employee of the Provincial Auditor and will observe 


nn nnn EEE EEyE SESE ESSE 
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Idem 


Record of oaths 


Cause for dismissal 


Benefits 


Pension plan 


Expert assistance 


and comply with the laws of Canada and Ontario and, except as I may be 
legally required, I will not disclose or give to any person any information or 
document that comes to my knowledge or possession by reason of my being 
an employee of the Office of the Auditor. 


So help me God. (Omit this line in an affirmation) 


(b) the following oath ofallegiance, in English or in French: 


Leen St MRE oa! Nand oi in ek bate Ce , do swear (or solemnly affirm) that 
I will be faithful and bear true allegiance to Her Majesty Queen Elizabeth the 
Second (or the reigning sovereign for the time being), her heirs and successors 
according to law. 


So help me God. (Omit this line in an affirmation) 


RS OMI eA 5 se a) 


(2) The Auditor may require any person or class of persons ap- 
pointed to assist the Auditor for a limited period of time or in respect of 
a particular matter to take and subscribe either or both of the oaths set 
out in subsection (1), R.S.O. 1990, c. A.35,s. 21 (2). 


(3) A copy of each oath administered to an employee of the Office 
of the Auditor under subsection (1) shall be kept in the file of the 
employee in the Office of the Auditor. R.S.O. 1990, c. A.35,s. 21 (3). 


(4) The failure of an employee of the Office of the Auditor to take 
and subscribe or to adhere to either of the oaths required by subsection 
(1) may be considered as cause for dismissal. R.S.O. 1990, c. A.35, 
SON 


22.—(1) The employee benefits applicable from time to time under 
the Public Service Act to civil servants who are not within a unit of 
employees established for collective bargaining under any Act apply or 
continue to apply, as the case may be, to the Auditor, the Assistant 
Auditor and to the full-time permanent and probationary employees of 
the Office of the Auditor and the Board or any person authorized by 
order of the Board may exercise the powers and duties of the Civil 
Service Commission and the Auditor or any person authorized in 
writing by the Auditor may exercise the powers and duties of a deputy 
minister under that Act in respect of such benefits. R.S.O. 1990, 

Cy Oost UNE 


(2) The Auditor and the Assistant Auditor are members of the 
Public Service Pension Plan. R.S.O. 1990, c. A.35,s. 22 (2). 


23. Subject to the approval of the Board, the Auditor from time to 
time may appoint one or more persons having technical or special 


eS 
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knowledge of any kind to assist the Auditor for a limited period of time 
or in respect of a particular matter and the money required for the 
purposes of this section shall be charged to and paid out of the Consoli- 
dated: Revenue Funds: RiS:@1:1990,(c:-Ac35,'s.. 23: 


24. The Auditor may delegate in writing to any other member of the 
Office of the Auditor authority to exercise any power or perform any 
duty of the Auditor other than reporting to the Assembly. R.S.O. 1990, 
C2Ai35;S) 24! 


25.—(1) An employee of the Office of the Auditor shall not, 


(a) bea candidate in a provincial or federal election or in an election 
for any municipal office including a local board of a municipality 
within the meaning of the Municipal Affairs Act; 


(b) solicit funds for a provincial, federal or municipal party or candi- 


date; or 


(©) associate his or her position in the Office of the Auditor with any 
political activity. R.S.O. 1990, c. A.35,s. 25 (1). 


(2) Contravention of any of the provisions of subsection (1) may be 


considered as cause for dismissal. R.S.O. 1990, c. A.35, s. 25 (2). 


26.—(1) The Auditor may make orders and rules for the conduct of 
the internal business of the Office of the Auditor and, after a hearing, 
may suspend, demote or dismiss any employee of the Office of the 


Auditor for cause. R.S.O. 1990, c. A.35, s. 26 (1). 


(2) The Public Service Act and regulations thereunder that apply in 
relation to suspension from employment pending an investigation and in 
relation to a hearing by a deputy minister or his or her delegate as to 
cause for dismissal, other than as to notice to the Civil Service Commis- 
sion, apply with necessary modifications where the Auditor is of the 
opinion that there may exist cause for the suspension without pay, demo- 
tion or dismissal of an employee of the Office of the Auditor, and, for the 
purpose, the Auditor shall be deemed to be a deputy minister. 

RS.) 1990;c A395; s. 26 (2). 


(3) A decision of the Auditor to demote, suspend or dismiss an em- 
ployee may be appealed by the employee, within fourteen days after the 
decision has been communicated to the employee to the Public Service 
Grievance Board established under the Public Service Act. R.S.O. 1990, 
Ces 2a 
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authorized to hear 
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Proceedings 
privileged 


Information 


confidential 
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the Auditor 
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Review by Board 


(4) The Public Service Grievance Board may hear and dispose of an 
appeal under this section and the provisions of the regulation under the 
Public Service Act that apply in relation to a grievance for dismissal apply 
with necessary modifications to an appeal under this section, and, for the 
purpose, the Auditor shall be deemed to be a deputy minister and the 
decision of the Public Service Grievance Board is final and the Public 
Service Grievance Board shall report its decision and reasons in writing to 


the Auditor and to the appellant. R.S.O. 1990, c. A.35, s. 26 (4). 


27.—(1) No proceedings lie against the Auditor, the Assistant Audi- 
tor, any person employed in the Office of the Auditor or any person 
appointed to assist the Auditor for a limited period of time or in respect 
of a particular matter, for anything he or she may do or report or say in 
the course of the exercise or the intended exercise of functions under this 
Act, unless it is shown that he or she acted in bad faith. R.S.O. 1990, 
CPEs Se 7) 


(2) The Auditor, the Assistant Auditor and each person employed in 
the Office of the Auditor or appointed to assist the Auditor for a limited 
period of time or in respect of a particular matter shall preserve secrecy 
with respect to all matters that come to his or her knowledge in the 
course of his or her employment or duties under this Act and shall not 
communicate any such matters to any person, except as may be required 
in connection with the administration of this Act or any proceedings 
under this Act or under the Criminal Code (Canada). R.S.O. 1990, 
CAS OESe 2A): 


28. A person or persons, not employed by the Crown or the Office of 
the Assembly, licensed under the Public Accountancy Act and appointed 
by the Board, shall examine the accounts relating to the disbursements of 
public money on behalf of the Office of the Auditor and shall report 
thereon to the Board and the chairman of the Board shall cause the 
report to be laid before the Assembly if it is in session or, if not, at the next 
session. R.S.0:1990, ¢:A.35,s. 28. 


29.—(1) The Auditor shall present annually to the Board estimates of 
the sums of money that will be required for the purposes of this Act. 
Res ONO Fer .6555129 (1). 


(2) The Board shall review and may alter as it considers proper the 
estimates presented by the Auditor, and the chair of the Board shall cause 
the estimates as altered by the Board to be laid before the Assembly and 
the Assembly shall refer the estimates laid before it to a committee of the 
Assembly for review. R.S.O. 1990, c. A.35,s. 29 (2). 
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Notice (3) Notice of meetings of the Board to review or alter the estimates 
presented by the Auditor shall be given to the chair and the vice-chair of 
the standing Public Accounts Committee of the Assembly and the chair 
and the vice-chair may attend at the review of the estimates by the Board. 


Bis. 1990 cs Ars eee (3). 
(4) The money required for the purposes of this Act, other than 


under sections 5 and 23, shall be paid out of the money appropriated 
therefor by the Legislature. R.S.O. 1990, c. A.35, s. 29 (4). 


Money 
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